April 8th, 2008

Google App Engine: Your Apps in the Cloud

Google just significantly raised the stakes in the platform-as-a-service market with tonight’s launch of Google App Engine, a scalable, fault-tolerant web application environment that lets developers run their own apps on Google’s infrastructure. Naturally the new platform leverages Google’s expertise in building web-scale services including Big Table-type storage.

While in many ways this service competes with Amazon’s suite of on-demand infrastructure APIs including S3 storage, EC2 hosting and the SimpleDB database, the approach is different. In Google’s model you get all of these services bundled together in one package. This is a plus if you want to run your entire app under one roof versus the lower-level, individual services in Amazon’s model.

At a high level there are five pieces to App Engine:

• The core fault-tolerant, scalable application environment that “runs reliably, even under heavy load and with large amounts of data.” A key part of this is the “sandbox” which isolates your application a secure environment that is “independent of the hardware, operating system and physical location of the web server.”
• A runtime environment for the Python programming language including libraries and modules. Again this is a difference from the Amazon model: here it’s not a set of web service APIs, but language-specific frameworks. While this limits the audience to Python developers for now, the underlying platform is language-neutral and support for more languages will be added in the future.
• A local development environment and SDK. When you sign-up for the service one of the first steps is to download the SDK and once installed, it creates a local development environment that from a code perspective mimics the runtime App Engine infrastructure. Developers can code locally against stubs and then upload.
• An adminstration console for app management, logging, and monitoring. The admin dashboard includes usage stats like requests per second, bytes received per second, CPU megacycles per second and quota denials per second.
• The BigTable model distributed data storage service: “Just as the distributed web server grows with your traffic, the distributed datastore grows with your data.” Note that as with Amazon’s SimpleDB, the App Engine datastore is not a traditional relational database. In this case it consists of “data objects, or entities, that have a kind and a set of properties. Queries can retrieve entities of a given kind filtered and sorted by the values of the properties. Property values can be of any of the supported property value types.”

One of the first questions from most developers will be: What’s the cost? Sign up is free and so is running your app as long as stay under quotas 500MB of storage, 200 million megacycles/day of CPU, and 10GB of total bandwidth. Google estimates this means there will be no cost for up to approximately 5 million pageviews a month. Once this initial preview period is over Google will introduce a billing model for additional resources at “competitive market prices.”

As you can see there’s a lot to this announcement. Google provides a lot of useful resources including What is Google App Engine?, an FAQ, and set of articles. Note that the preview release is limited to the first 10,000 developers that sign-up.

And finally, a note on today’s launch itself: just as they’d done with their OpenSocial announcement last fall, Google launched App Engine at a Campfire on Google’s Mountain View campus. For details on the event itself check-out TechCrunch’s coverage including videos from Robert Scoble.

Share This

Posted by John Musser as Google, Infrastructure at 1:34 AM | 4 Comments »

March 27th, 2008

Amazon Enhances EC2, Embraces Failure

Amazon’s web services team has just announced two enhancements to their successful EC2 cloud computing service that makes it even more desirable and useful by making it more fault-tolerant: the ability to place instances in multiple locations and “Elastic IP addresses”.

As you can see below, both of these are key pieces of the puzzle in how to build dynamic systems in the cloud that can deliver both scalability and fault-tolerance. As O’Reilly Radar’s Jesse Robbins points out “Datacenters and geographic regions are Single Points of Failure (SPOF) too. Failure Happens, and it’s far better (and cheaper) to build services that are resilient to failure than to try to prevent them from happening. This is a big step in the right direction.”

• Elastic IP Addresses are static IP addresses designed for dynamic cloud computing, and now make it easy to host web sites, web services and other online applications in Amazon EC2. Elastic IP addresses are associated with your AWS account, not with your instances, and can be programmatically mapped to any of your instances. This allows you to easily recover from instance and other failures while presenting your users with a static IP address.
• Availability Zones give you the ability to easily and inexpensively operate a highly available internet application. Each Amazon EC2 Availability Zone is a distinct location that is engineered to be insulated from failures in other Availability Zones. Previously, only very large companies had the scale to be able to distribute an application across multiple locations, but now it is as easy as changing a parameter in an API call. You can choose to run your application across multiple Availability Zones to be prepared for unexpected events such as power failures or network connectivity issues, or you can place instances in the same Availability Zone to take advantage of free data transfer and the lowest latency communication.

Amazon’s blog post points to a useful series of new articles on EC2 by the team at RightScale: DNS and Elastic IPs and how they come in to play when upgrading a server, setting up a fault-tolerant site using the Availability Zones, and now support the new Elastic IP and Availability Zone. Includes good details, walk-throughs and illustrations like below.

Share This

Posted by John Musser as Amazon, Infrastructure at 11:43 AM | 1 Comment »

March 13th, 2008

Microsoft’s Data Stack Goes Up to the Cloud

Building on their flurry of announcements around last week’s MIX08 event, Microsoft’s Live Platform Services GM, George Moore has just outlined in the dev.live blog how Microsoft plans to unify their storage solutions from on premise all the way up to cloud storage. As you can see below, the strategy spans a range of products and services. As a driving theme, the post makes reference to Ray Ozzie’s MIX08 keynote and ideas around having a connected “mesh” of synchronized devices and services.

As George describes “For the first time ever we have a unified protocol and developer tooling story across most of our major storage products from Microsoft”. Microsoft provides the following table to help summarize the layers on this stack. The four layers build up from Microsoft-specific products and services, the open standard protocols, developer tools, and at the highest level is a synchronization infrastructure.

Technically, two key pieces of the puzzle are: a) using Atom as the “unified on-the-wire format” with AtomPub as the “unified protocol”, and b) “a set of URI namespace conventions to address scalar values and feed-of-feed hierarchy navigation which also work uniformly across the above storage products and services, regardless of the top level DNS address of the underlying service.” The latter are outlined in this table:

Earlier this month ZDNet’s Mary Jo Foley wrote about Microsoft working to stitch together a comprehensive story on unified storage. It’s safe to assume there will be additional Microsoft announcements before long as more pieces of the Live Services puzzle fall into place (23 Live APIs so far).

Share This

Posted by John Musser as Infrastructure, Microsoft at 1:26 AM | No Comments »

February 7th, 2008

Scale Facebook Apps With Amazon Services

Amazon and Facebook have clearly been on the leading edge of two forces driving the web as platform - cloud computing and exploitation of the social graph. Now these two companies have announced a partnership that lets Facebook developers leverage both innovations by giving developers a path to employ the Amazon cloud computing model based on the EC2 and S3 APIs. This has the ability to provide an infrastructure for developers who dream of building a wildly viral Facebook application and worry about how to handle the growth.

Amazon’s EC2 provides the metered compute capacity for creating virtual instances, and S3 the storage component. The partnership with Facebook doesn’t extend either of their respective APIs, but it offers a a set of resources including a step-by-step process to launch a Facebook app using the Amazon platform.

Facebook developers had already one solid option with one-year-free hosting from Joyent which partnered with Facebook in December and it’s likely we’ll see a variety of such app infrastructure services appearing in the next few months.

Share This

Posted by John Musser as Amazon, Facebook, Infrastructure, Social at 1:48 AM | No Comments »

January 31st, 2008

Amazon Web Services Make Earnings News

The just released Amazon Q4 2007 earnings report, besides showing that the company doubled profits this quarter, had a couple very interesting notes related to their growing suite of web services. The first is that bandwidth usage from their web services exceeds that of their web sites:

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

This is somewhat reminiscent of the point at which Salesforce.com began to process nearly as many transactions via their APIs as through their application proper.

And the second piece of news from that division is that “Over 330,000 developers have registered to use Amazon Web Services (AWS), up more than 30,000 from last quarter.” These two details go hand in hand with more developers signed-up and more usage of the core pay-as-you-go infrastructure services.

You can read more on the Amazon results from BusinessWeek’s Rob Hof, paidContent, TechCrunch and Laurie Flynn at the New York Times.

Share This

Posted by John Musser as Amazon, Infrastructure, Money at 1:21 AM | 2 Comments »

January 2nd, 2008

Amazon DevPay: Resell Your Web Services

The Amazon Web Services team just ended an impressive year with one last innovation: Amazon DevPay. DevPay builds on Amazon’s strengths in running both online shopping services and online web services to create a business infrastructure to support developers using their web services like S3 and EC2. It helps simplify the process of billing and tracking for apps that use these pay-per-use Amazon APIs, essentially enabling a reseller model (DevPay also includes it’s own licensing API and you can see our a API profile here).

As they describe on the AWS blog:

This new service allows entrepreneurial developers to wrap their own business models around Amazon S3 and Amazon EC2, taking advantage of Amazon’s existing customer base and billing infrastructure. With DevPay, developers can focus on being creative and innovative while dispatching the less-than-glamorous aspects of dealing with bank accounts, credit cards, and so forth to us.

Developers use DevPay’s web-based registration interface to create pricing plans for their applications, monitor customer signups, and track usage. The developer’s customers use another web-based interface to sign up and enter payment information for the applications that they wish to use.

You can think of DevPay as an enabling technology for our other services.

There were some initial questions about the differences between the Amazon Flexible Payments API and DevPay, which the AWS team has clarified here.

Using Amazon Flexible Payments Service (Amazon FPS), developers can accept payments on websites. It has several innovative features, including support for micropayments.

Amazon DevPay instruments two Amazon Web Services to enable a new sort of Software as a Service. Amazon DevPay supports applications built on Amazon S3 or Amazon EC2 by allowing you to “resell” applications built on top of one of these services. You determine the retail price, which is a mark-up above Amazon’s base price. Customers pay for your application by paying Amazon. We deduct the base price plus a small commission; then deposit the rest into your Amazon account.

It’s another intriguing building block in the web services infrastructure stack. The cost? “Your customers will be billed for usage of their DevPay-powered applications on the first day of each month. We will then deduct a 3% fee plus another 30 cents, and deposit the remainder in your DevPay account.”

One of the first S3 customers to begin using this service is Amanda Enterprise, a supported version of the Amanda open source backup and recovery tool which now use DevPay along with Amazon S3 to backup, archive and retrieve customer data. See Amanda profile here.

Looks like SmugMug might start using it soon as well.

Share This

Posted by John Musser as Amazon, Infrastructure, Money at 12:28 AM | 2 Comments »

December 31st, 2007

Amazon’s EC2 Open Source Firefox Plugin

Developers using Amazon’s EC2 API might find this interesting: Amazon has created an open source project on SourceForge for ElasticFox, their Firefox extension that lets you create and manage EC2 instances from a GUI in the browser. The utility lets you launch AMIs (Amazon Machine Instances) and then find, control, replicate and shutdown running instances. A handy right-click menu lets you quick launch more like ones already running. Having the source available will let developers extend this as needed. For more see the Amazon AWS blog and at our mashup profile here.

Share This

Posted by John Musser as Amazon, Infrastructure, Source at 12:03 AM | 2 Comments »