Perspectives - Netflix on AWS

Monday, May 31, 2010

I did a talk at the Usenix Tech conference last year, Where does the Power Go in High Scale Data Centers. After the talk I got into a more detailed discussion with many folks from Netflix and Canada’s Research in Motion, the maker of the Blackberry. The discussion ended up in a long lunch over a big table with folks from both teams. The common theme of the discussion was predictably, given the companies and folks involved, innovation in high scale service and how to deal with incredible growth rates. Both RIM and Netflix are very successful and, until you have experienced and attempted to manage internet growth rates, you really just don’t know. I'm impressed with what they are doing. Growth brings super interesting problems and I learned from both and really enjoyed spending time with them.

I recently came across an interesting talk by Santosh Rau, the Netflix Cloud Infrastructure Engineering Manager. The fact that Netflix actually has a Cloud Infrastructure engineering manager is what caught my attention. Netflix continues to innovate quick and is moving fast with cloud computing.

My notes from Rau’s talk:

· Details on Netflix

o More than 10m subscribers

o Over 100,000 DVD titles

o 50 distribution centers

o Over 12,000 instant watch titles

· Why is Netflix going to the cloud

o Elastic infrastructure

o Pay for what you use

o Simple to deploy and maintain

o Leverage datacenter geo-diversity

o Leverage application services (queuing, persistence, security, etc.

· Why did Netflix chose Amazon Web Services

o Massive scale

o More mature services

o Thriving, active developer community of over 400,000 developers with excellent support

· Netflix goals for move to the cloud:

o Improved availability

o Operational simplicity

o Architect to exploit the characteristic of the cloud

· Services in cloud:

o Streaming control service: stream movie content to customers

§ Architecture: Three Netflix services running in EC2 (replication, queueing, and streaming) with inter-service communication via SQS and persistent state in SimpleDB.

§ Good cloud workload in that usage can vary greatly and there is value in having regional data centers and a better customer experience is possible by streaming content from locations near users

o Encoding Service: Encodes movies in format required by diverse set of supported devices.

§ Good cloud workload in that its very computational intense and as new formats are introduced, massive encoding work needs to be done and there is value in doing it quickly (more servers for less time).

o AWS Services used by Netflix

§ Elastic compute Cloud

§ Elastic Block Storage

§ Simple Queuing Service

§ SimpleDB

§ Simple Storage Service

§ Elastic Load Balancing

§ Elastic MapReduce

o Developer Challenges:

§ Reliability and capacity

§ Persistence strategy

· Oracle on EC2 over EBS vs MySQL vs SimpleDB

· SimpleDB: Highly available replicating across zones

· Eventually consistent (now supports full consistency (I love eventual consistency but…)

§ Data encryption and key management

§ Data replication and consistency

Predictably, the talk ended with “Netflix is hiring” but, in this case, it is actually worth mentioning. They are doing very interesting work and moving lightening fast. RIM is hiring as well: http://www.rim.com/careers/index.shtml.

The slides for the talk are at: slideshare.

--jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

Monday, May 31, 2010 5:59:29 AM (Pacific Standard Time, UTC-08:00)

Comments [4] - Trackback
Services

Tuesday, June 01, 2010 12:56:16 AM (Pacific Standard Time, UTC-08:00)

The Netflix slides are very interesting.

What I wonder is whether deploying Netflix services on Amazon infrastructure is such a wise decision strategically. After all Netflix and Amazon could well turn into competitors in several areas, e.g. video-on-demand. Amazon could potentially terminate the Netflix agreement (within the terms of the service agreement) and severely disrupt the Netflix operations.

Jan Miczaika

Tuesday, June 01, 2010 4:31:17 AM (Pacific Standard Time, UTC-08:00)

All companies of reasonable size or beyond either have areas of competition with each other or will. Any infrastructure company that doesn't support competitors as well as they support themselves is making a short term decision that will kill there infrastructure business and this probably wouldn't be a slow process. These businesses are built on trust. Unless all customers are 100% convinced they are getting the best service possible regardless of competitive position, none will trust the infrastructure provider.

Amazon's Jeff Bezos has used Netflix in public talks as an example of the complex relationships that can emerge in the infrastructure business. He's made it very clear that Netflix gets the same high quality service that all AWS customers get and that will always be true independent of competitive positioning with the rest of Amazon. He says he would also be very happy to host the next big retail competitor for Amazon.com and explains how Amazon has always run the busines this way. For example, 30% of the Amazon retail sales business comes from non-Amazon merchants selling on the Amazon.com infrastructure in competition with the Amazon.com retail business. Complex to be sure but good for customers, competitors, and even Amazon.

I can't see competitive position with Amazon ever influencing how well a company is served by AWS.

--jrh

James Hamilton

Monday, June 07, 2010 8:03:04 AM (Pacific Standard Time, UTC-08:00)

James, as long as Amazon is growing constantly and has a great culture of smart people it will work out fine. Should the going ever get tough (what I of course don't hope) these principles may be thrown overboard. It would not be the first time companies sacrifice long-term values for short-term profits.

This is all very hypothetical. Still, for strategic long-term planning, I believe it should be taken into account.

Jan Miczaika

Monday, June 07, 2010 8:33:22 AM (Pacific Standard Time, UTC-08:00)

Jan, it is inarguably true that there have been instance of previously good companies making incredibly short sited decisions. Its happened before and it'll happen again.

The point I'm making is not that there exists any company that is incapable of damn dumb decision. My point is that the cloud computing model is a huge win economically. However, I agree with you that no company is assured to be great, customer focused, and thinking clearly forever. Disasters can happen. That's why I would never do business with a cloud provider that didn't have great support for export of LARGE amounts of data cost effectively. Its super important that the data not be locked in. I don't care so much about the low level control plane program model -- I can change that. But its super important that the data can be moved to another service easily. And, this export service has to be cheap and there is no way I would use the network for very high scale data movements. You have to assume that the data is going to keep growing and so its physical media export that you want. Recall Andrew Tenenbaum's "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway".

I'm saying you need to use cloud computing but I'm not saying you should trust one company to be the right answer for ever. Don't step in without a good quality export service based upon physical media at a known and reasonably price.

--jrh
jrh@mvdirona.com

James Hamilton

Comments are closed.