Wow, the pace is starting to pick up in the service platform world. Google announced their long awaited entrant with Google Application Engine last Monday, April 7th. Amazon announced the SimpleDB to answer the largest requirement they were hearing from AWS customers: persistent, structured storage. Yesterday, another major step was made with Werner Vogles announcing availability of persistent storage for EC2.
Persistence for EC2 is a big one. I’ve been amazed at how hard customers were willing to work to get persistent storage in EC2. The most common trick is to periodically snapshot the up to 160GB of ephemeral state allocated to each Amazon EC2 instance to S3. This does work but is very clunky and looses all between the last snap shot and non-orderly shutdown is a bit nasty. A solution I like is a replicated block storage layer like DRBD. One innovative solution to all EC2 state being transient is to use DRDB to maintained a replicated file system between two EC2 instances. Not bad – in fact I really like it but it’s hard to set up and, last time I checked, only supported 2-way redundancy when 3 is where you want to be when using commodity hardware.
It appears the solution is (nearly) here with EC2 persistence. The model they have chosen storage volume as the abstraction. Any number of storage volumes can be created in sizes of up to 1TB. Each storage volume is created in a developer specified availability zone and each volume supports snapshots to S3. A volume can be created from a snap-shot. The supported redundancy and recovery models were not specified but I would expect that they are using redundant, commodity storage. Werner did say it was file system semantics which I interpret as cached, asynchronous write with optional application controlled write through/flush. It is not clear if shared volumes are supported (multiple EC2 instances accessing the same volume).
Another blog entry from Amazon “demo’s” Usage: I spent some time experimenting with this new feature on Saturday. In a matter of minutes I was able to create a pair of 512 GB volumes, attach them to an EC2 instance, create file systems on them with mkfs, and then mount them. When I was done I simply unmounted, detached, and then finally deleted them.
Unfortunately, persistent storage for EC2 won’t be available until “later this year” but it looks like a good feature that will be well received by the development community.
Update: This may be closer to beta than I thought. I just (5:52am 4/14) reciveved a limited beta invitation.
Thanks to David Golds and Dare Obasanjo for sending pointers my way.
James Hamilton, Windows Live Platform Services Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052 W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com
Disclaimer: The opinions expressed here are my own and do not
necessarily represent those of current or past employers.