Building Scalable Web Apps with Google App Engine was presented by Brett Slatkin of Goolgle at Google I/O 2008. The link above points to the video but Todd Hoff of High Scalability summarized the presentation in a great post Numbers Everyone Should Know.
The talk mostly focused on the Google App Engine and how to use it. For example, Brett shows how to implement a scalable counter and (nearly) ordered comments using App Engine Megastore. For the former, shard the counter to get write scale and sum them on read.
Also included in the presentation where some general rule of thumb from Jeff Dean of Google. Rules of Thumb are good because they tell us what to expect and, when we see something different, they tell us to pay attention and look more closely. When we see an exception, either our rule of thumb has just been proven wrong and we learned something. Or the data we’re looking at is wrong and we need to dig deeper. Either one is worth noticing. I use Rules of Thumb all the time not as way of understanding the world (they are sometimes wrong) but as a way of knowing where to look more closely.
Check out Toldd’s post: http://highscalability.com/numbers-everyone-should-know.
–jrh
James Hamilton, Amazon Web Services
1200, 12th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com
H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com
I’ve made similar suggestion in the past and I still think it’s a good idea: some customers want low-cost, low access speed storage while others need high-access speed storage (and there is a higher cost in providing that). I agree.
–jrh
I am checking out this new Google App Engine billing system and I wonder, why does storage have to cost the same for fast access scalable web code for web pages and for storing a huge amounts of videos or other large backup files which aren’t necesarilly needed to be on fast-access storage servers, which may in some cases never even be downloaded but just need to be there on the cloud for backup purposes..
I posted a comment here a couple months ago about complete offline offpower TB hard drives using some type of mechanical or electronic switch. That there might be some delay in time to access those files, but then just like rapidshare.com (or how those ad-supported file storage sites are called), you could display a counter such as "Your file will be available in 30 seconds" and it would count down in realtime. The time to access could depend on realtime traffic to that node where one of your two hard drives are located (for reasonably good reliability files can be stored on two hard drives at two different places in the cloud).
Basically what I am looking for is $100/TB/year storage on the cloud for my huge collection of videos. Not $1800/TB/year which is the current Google App Engine storage cost would obviously not be an option for people like me who just would like to store High Definition videos that are not obviously all going to be for very high traffic commercial use.
I would have thought that since bandwidth has a cost, that higher bandwidth usage on certain data would then also cover the higher cost of the storage for popular data. Perhaps Google could differentiate storage, bandwidth and CPU cycle costs for high access web content and another price for large files which may or may not require a lot of downloads.
Another thing I really am hoping for, is a user friendly way to install PHP applications on App Engine soon such as WordPress, phpbb3, mediawiki and more things like that. It’d be really great if Google could take over the hosting of those things, easily integrate them all with Open Social authentication systems (so users can just log onto them using their Google accounts), easily build Google App plugins such as Google Checkout integration, integrating the Youtube API and more.