The New Amazon S3 API

John Musser, March 14th, 2006

Comments(2)

The big API news today is that the web as platform now has a new world-class storage system designed specifically for developers: Amazon S3, their Simple Storage Service. Amazon has basically taken the online storage infrastructure behind their core online services and provided a public, fee-based interface to it. There’s now a viable “storage cloud” out there for you to use.

Storage isn’t sexy, but as anyone in IT can tell you: almost nothing’s more important than a reliable storage infrastructure. So it’s good news for web platform developers to have this class of storage service available for whatever they need it for. Here is a key part of your virtual data center.

Yesterday I spoke with Adam Selipsky, Amazon’s Web Services VP of Product Management, and Dave Barth, the Product Manager for Amazon S3. They emphasized that that API was designed to a focus on a core set of functions and do them well.

A couple of examples cited in Amazon’s announcement may trigger ideas on what it might be used for:

• University of California Berkeley “Stardust@Homeâ€? Team: Saved money by using Amazon S3 to store and deliver the 60,000 images that represent the data collected from their dust particle aerogel experiment. These images will be delivered to 100,000 volunteers around the world who scan the images looking for dust particles from comet Wild2. Since this phase of the project needs the storage for just a few months they were able to ramp up and back down with less investment.
• CastingWords: This small startup offering podcast transcription services were able to save effort and time by using S3 to store and retrieve the original audio files and the transcribed texts.
• BitTorrent: Oh yes, and there is even a section in the online documentation on how to use S3 for BitTorrent files. For publishers of large, popular files the amount of data actually supplied by S3 can be substantially lower than what it would have been serving the same clients via client/server download. Less data transferred means lower costs for the publisher of the object.

Note also what this isn’t: it’s not an online storage service in the same vein as what you’d get from box.net or their competitors. It is a service for developers and not end users. Here, not only there is no friendly user interface, there is no UI at all. For now you can only get to it via code. It is completely and solely for developers to build tools and applications on top of. Thus eventually there will be pretty tools, OS desktop integration, and so on. Amazon pioneered a bit of this API-only model with their unique Mechanical Turk API.

Nor is it like hacks people have built on top of GMail, those are just handy hacks for some power users (speaking of which, it could be that the often rumored GDrive will provide a comparable API but that remains to be seen and Amazon’s API is live now). And while Amazon is first out of the gate in this league there will certainly be some serious competition from the usual suspects.

What does it cost? It’s $0.15 per gigabyte of storage per month and $0.20 per gigabyte of data transferred. This model is nice because you only pay for what you use, no more, no less, with no setup fees or monthly minimums.

Technical details? It supports both REST and SOAP, but both score high on simplicity. No fancy extras, just core file services: store, retrieve, delete. No nonessential services. The REST API maps these functions directly to the core HTTP RFC 2616 requests: GET, PUT and DELETE. Everything is an Object which is opaque to Amazon. For more details see their site or the Amazon S3 entry here at ProgrammableWeb.

How reliable is the service and is there an SLA? Amazon says this is a four-nines service with 99.99% uptime and that you can trust it because it’s the same platform they run amazon.com on. There is no written SLA in the same form you would get from your ISP. For some this might be an issue, although certainly many SLAs have no teeth anyway. Given the issues that Salesforce.com has encountered, reliability is a key success factor for Amazon. It would be nice someday if they have some form of a service status dashboard similar to the Salesforce.com’s status.salesforce.com/.

The idea that secure, reliable storage for any given application will just “be there up in the cloud” is powerful and this is a big step in that direction.