Perspectives - Services

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Amazon DynamoDB: NoSQL in the Cloud

James Hamilton — Wed, 18 Jan 2012 21:00:06 GMT

Finally! I’ve been dying to talk about DynamoDB since work began on this scalable, low-latency, high-performance NoSQL service at AWS. This morning, AWS announced availability of DynamoDB: Amazon Web Services Launches Amazon DynamoDB – A New NoSQL Database Service Designed for the Scale of the Internet.

In a past blog entry, One Size Does Not Fit All, I offered a taxonomy of 4 different types of structured storage system, argued that Relational Database Management Systems are not sufficient, and walked through some of the reasons why NoSQL databases have emerged and continue to grow market share quickly. The four database categories I introduced were: 1) features-first, 2) scale-first, 3) simple structure storage, and 4) purpose-optimized stores. RDBMS own the first category.

DynamoDB targets workloads fitting into the Scale-First and Simple Structured storage categories where NoSQL database systems have been so popular over the last few years. Looking at these two categories in more detail, Scale-First is:

Scale-first applications are those that absolutely must scale without bound and being able to do this without restriction is much more important than more features. These applications are exemplified by very high scale web sites such as Facebook, MySpace, Gmail, Yahoo, and Amazon.com. Some of these sites actually do make use of relational databases but many do not. The common theme across all of these services is that scale is more important than features and none of them could possibly run on a single RDBMS. As soon as a single RDBMS instance won’t handle the workload, there are two broad possibilities: 1) shard the application data over a large number of RDBMS systems, or 2) use a highly scalable key-value store.

And, Simple Structured Storage:

There are many applications that have a structured storage requirement but they really don’t need the features, cost, or complexity of an RDBMS. Nor are they focused on the scale required by the scale-first structured storage segment. They just need a simple key value store. A file system or BLOB-store is not sufficiently rich in that simple query and index access is needed but nothing even close to the full set of RDBMS features is needed. Simple, cheap, fast, and low operational burden are the most important requirements of this segment of the market.

More detail at: One Size Does Not Fit All.

The DynamoDB service is a unified purpose-built hardware platform and software offering. The hardware is based upon a custom server design using Flash Storage spread over a scalable high speed network joining multiple data centers.

DynamoDB supports a provisioned throughput model. A DynamoDB application programmer decides the number of database requests per second their application should be capable of supporting and DynamoDB automatically spreads the table over an appropriate number of servers. At the same time, it also reserves the required network, server, and flash memory capacity to ensure that request rate can be reliably delivered day and night, week after week, and year after year. There is no need to worry about a neighboring application getting busy or running wild and taking all the needed resources. They are reserved and there whenever needed.

The sharding techniques needed to achieve high requests rates are well understood industry-wide but implementing them does take some work. Reliably reserving capacity so it is always there when you need it, takes yet more work. Supporting the ability to allocate more resources, or even less, while online and without disturbing the current request rate takes still more work. DynamoDB makes all this easy. It supports online scaling between very low transaction rates to applications requiring millions of requests per second. No downtime and no disturbance to the currently configured application request rate while resharding. These changes are done online only by changing the DynamoDB provisioned request rate up and down through an API call. >

In addition to supporting transparent, on-line scaling of provisioned request rates up and down over 6+ orders of magnitude with resource reservation, DynamoDB is also both consistent and multi-datacenter redundant. Eventual consistency is a fine programming model for some applications but it can yield confusing results under some circumstances. For example, if you set a value to 3 and then later set it to 4, then read it back, 3 can be returned. Worse, the value could be set to 4, verified to be 4 by reading it, and yet 3 could be returned later. It’s a tough programming model for some applications and it tends to be overused in an effort to achieve low-latency and high throughput. DynamoDB avoids forcing this by supporting low-latency and high throughout while offering full consistency. It also offers eventual consistency at lower request cost for those applications that run well with that model. Both consistency models are supported.

It is not unusual for a NoSQL store to be able to support high transaction rates. What is somewhat unusual is to be able to scale the provisioned rate up and down while on-line. Achieving that while, at the same time, maintaining synchronous, multi-datacenter redundancy is where I start to get excited.

Clearly nobody wants to run the risk of losing data but NoSQL systems are scale-first by definition. If the only way to high throughput and scale, is to run risk and not commit the data to persistent storage at commit time, that is exactly what is often done. This is where DynamoDB really shines. When data is sent to DynamoDB, it is committed to persistent and reliable storage before the request is acknowledged. Again this is easy to do but doing it with average low single digit millisecond latencies is both harder and requires better hardware. Hard disk drives can’t do it and in-memory systems are not persistent so flash memory is the most cost effective solution.

But what if the server to which the data was committed fails, or the storage fails, or the datacenter is destroyed? On most NoSQL systems you would lose your most recent changes. On the better implementations, the data might be saved but could be offline and unavailable. With dynamoDB, if data is committed just as the entire datacenter burns to the ground, the data is safe, and the application can continue to run without negative impact at exactly the same provisioned throughput rate. The loss of an entire datacenter isn’t even inconvenient (unless you work at Amazon :-)) and has no impact on your running application performance.

Combining rock solid synchronous, multi-datacenter redundancy with average latency in the single digits, and throughput scaling to the millions of requests per second is both an excellent engineering challenge and one often not achieved.

More information on DynamoDB:

· Press Release: http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1649209&highlight=

· DynamoDB detail Page: http://aws.amazon.com/dynamodb/

· DynamoDB Developer Guide: http://docs.amazonwebservices.com/amazondynamodb/latest/developerguide/

· Blog entries:

o Werner: http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html

o Jeff Barr: http://aws.typepad.com/aws/2012/01/amazon-dynamodb-internet-scale-data-storage-the-nosql-way.html

· DynamoDB Frequently Asked Questions: http://aws.amazon.com/dynamodb/faqs/

· DynamoDB Pricing: http://aws.amazon.com/dynamodb/pricing/

· GigaOM: http://gigaom.com/cloud/amazons-dynamodb-shows-hardware-as-mean-to-an-end/

· eWeek: http://www.eweek.com/c/a/Database/Amazon-Web-Services-Launches-DynamoDB-a-New-NoSQL-Database-Service-874019/

· Seattle Times: http://seattletimes.nwsource.com/html/technologybrierdudleysblog/2017268136_amazon_unveils_dynamodb_databa.html

Relational systems remain an excellent solution for applications requiring Feature-First structured storage. AWS Relational Database Service supports both the MySQL and Oracle and relational database management systems: http://aws.amazon.com/rds/.

Just as I was blown away when I saw it possible to create the world’s 42^nd most powerful super computer with a few API calls to AWS (42: the Answer to the Ultimate Question of Life, the Universe and Everything), it is truly cool to see a couple of API calls to DynamoDB be all that it takes to get a scalable, consistent, low-latency, multi-datacenter redundant, NoSQL service configured, operational and online.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Global Netflix Platform: Large Scale Java PaaS Running on AWS

James Hamilton — Tue, 22 Nov 2011 21:09:55 GMT

Netflix is super interesting in that they are running at extraordinary scale, are a leader in the move to the cloud, and Adrian Cockcroft, the Netflix Director of Cloud Architecture, is always interesting in presentations. In this presentation Adrian covers similar material to his HPTS 2011 talk I saw last month.

His slides are up at: http://www.slideshare.net/adrianco/global-netflix-platform and my rough notes follow:

· Netflix has 20 milion streaming members

o Currently in US, Canada, and Latin America

o Soon to be in UK and Ireleand

· Netflix is 100% public cloud hosted

· Why did Netflix move from their own high-scale facility to a public cloud?

o Better business agility

o Netflix was unable build datacenters fast enough

o Capacity growth was both accelerating and unpredictable

o Product launch spikes require massive new capacity (iPhone, Wii, PS3, & Xbox)

Netflix grew 37x from Jan 2010 through Jan 2011

· Why did Netflix choose AWS as their cloud solution?

o Chose AWS using Netflix own platform and tools

o Netflix has unique platform requirements and extreme scale needing both agility & flexibility

o Chose AWS partly because it was the biggest public cloud

§ Wanted to leverage AWS investment in features and automation

§ Wanted to use AWS availability zones and regions for availability, scalability, and global deployment

§ Didn’t want to be the biggest customer on a small cloud

o But isn’t Amazon a competitor?

§ Many products that compete with Amazon run on AWS

§ Netflix is the “poster child” for the AWS Architecture

§ One of the biggest AWS customers

§ Netflix strategy: turn competitors into partners

o Could Netflix use a different cloud from AWS

§ Would be nice and Netflix already uses 3 interchangeable CDN vendors

§ But no one else has the scale and features of AWS

· “you have to be tall to ride this ride”

· Perhaps in 2 to 3 years?

o “We want to use cloud, we don’t want to build them”

§ Public clouds for agility and scale

§ We use electricity too but we don’t want to build a power station

§ AWS because they are big enough to allocated thousands of instances per hour when needed

o Netflix Global PaaS

§ Supports all AWS Availability Zones and Regions

§ Supports multiple AWS accounts (test, prod, etc.)

§ Supports cross Regions and cross account data replication & archiving

§ Supports fine grained security with dynamic AWS keys

§ Autoscales to thousands of instances

§ Monitoring for millions of metrics

o Portals and explorers:

§ Netflix Application Console (NAC): Primary AWS provisioning & config interface

§ AWS Usage Analyzer: cost breakdown by application and resource

§ SimpleDB Explorer: browse domains, items, attributes, values,…

§ Cassandra Explorer: browse clusters, keyspaces, column families, …

§ Base Service Explorer: browse endpoints, configs, perf metrics, …

o Netflix Platform Services:

§ Discovery: Service Register for applications

§ Introspections: Endpoints

§ Cryptex: Dynamic security key management

§ Geo: Geographic IP lookup engine

§ Platform Serivce: Dynamic property configuration

§ Localization: manage and lookup local translations

§ EVcache: Eccetric Volatile (mem)Cached

§ Cassadra: Persistence

§ Zookeeper: Coordination

o Netflix Persistence Services:

§ SimpleDB: Netflix moving to Cassandra

· Latencies typically over 10msec

§ S3: using the JetS3t based interface with Netflix changes and updates

§ Eccentric Volatile Cache (evcache)

· Discovery aware memcached based backend

· Client abstractions for zone aware replication

· Supports option to write to all zones, fast read from local

· On average, latencies of under 1 msec

§ Cassandra

· Chose because they value availability over consistency

· On average, latency of “few microseconds”

§ MongoDB

§ MySQti: supports hard to scale, legacy, and small relational models

o Implemented a Multi-Regional Data Replication system:

§ Oracle to SimpleDB and queued reverse path usingj SQS

o High Availability:

§ Cassandra stores 3 local copies, 1 per availability zone

§ Each AWS availability zone is a separate building with separate power etc. but still fairly close together so synchronous access is practical

§ Synchronous access, durable, and highly available

Adrian’s slide deck is posted at: http://www.slideshare.net/adrianco/global-netflix-platform.

--jrh

James Hamilton

From Perspectives.

Free Lessons in Industrial Design & Customer Experience

James Hamilton — Fri, 18 Nov 2011 15:44:52 GMT

I seldom write consumer product reviews and this blog is about the furthest thing from a consumer focused site but, every so often, I come across a notable tidbit that is worthy of mention. A few weeks ago, it was Sprint unilaterally changing the terms of their wireless contracts (Sprint is Giving Free Customer Service Lessons). It just seemed a sufficiently confused decision that it was worthy of mention.

Here’s one that just nails it on the other side of the equation by obsessing over the customer experience: Roku. I’ve long known about Roku but I’m not a huge TV watcher so I’ve only been peripherally interested in the product. But we are both Netflix and Amazon Prime Instant Video customers and Roku supports both. And the entry level Roku streaming appliance is only $49 so we figured let’s give it a try. It actually ended up a bit more than $49 in that we first managed to upsell ourselves to a $59 Roku 2 to get HD, and then to a $79 device to get 1080P and then to a $99 device to het 1080P HD with a hardwired Ethernet connection. So we ended up with a $100 device. I think $50 is close to where this class of devices needs to end up but $100 is reasonable as well.

The device is amazing and shows what can be done with a focus on clean industrial design. It is incredibly small at only 3” square. I plugged it in, it booted up, updated its software, found its remote, upgraded the software on the remote and went live without any user interaction. I setup a Roku account, linked my Amazon account for access to Prime Instant Video, linked our Netflix account and it was ready to go.

The device is tiny, produces close to no heat, you don’t have to read the manual, the user interface is clean and notable for its snappiness. I expected a sluggish UI as many companies scrimp on processing power to get costs down but it is very snappy. In fact Netflix on a Roku is faster than the same support on an Xbox. The UI is clean, simple, snappy, and very elegant.

I love where consumer appliances are heading: simple, cheap, dedicated, purpose-build devices with clean user interfaces, and the hybrid delivery model where the user interface is delivered by the appliance but most of the functionality is hosted in the cloud. The combination of cheap microelectronics, open source operating systems, and cloud hosting allows incredibly high function devices to be delivered at low cost.

The Kindle Fire takes the hybrid cloud connected model a long way where the Fire’s Silk browser UI runs directly on the device close to the user where it can be highly interactive and responsive. But the power and network-bandwidth hungry browser backend is hosted on Amazon EC2 where connectivity is awesome and compute power is not battery constrained. I love the hybrid model and we are going to see more and more devices delivering a hybrid user experience where the compute intensive components are cloud hosted and user interface is in the device. My belief is that this is the future of consumer electronics and, as prices drop to the $30 to $50 range, everyone will have 10s of these special-purpose, cloud-connected devices.

For the first time in my life, I’m super interested in consumer devices and the possibilities of what can be done in the hybrid cloud-connected appliance model.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

42: The Answer to the Ultimate Question of Life, the Universe, and Everything

James Hamilton — Tue, 15 Nov 2011 18:21:38 GMT

Yesterday the Top 500 Supercomputer Sites was announced. The Top500 list shows the most powerful commercially available supercomputer systems in the world. This list represents the very outside of what supercomputer performance is possible when cost is no object. The top placement on the list is always owned by a sovereign funded laboratory. These are the systems that only government funded agencies can purchase. But they have great interest for me because, as the cost of computing continues to fall, these performance levels will become commercially available to companies wanting to run high scale models and data intensive computing. In effect, the Top500 predicts the future so I’m always interested in the systems on the list.

What makes this list of the fastest supercomputers in the world released yesterday particularly unusual can be found at position #42. 42 is an anomaly of the first order. In fact, #42 is an anomaly across enough dimensions that its worth digging much deeper.

Virtualization Tax is Now Affordable:

I remember reading through the detailed specifications when the Cray 1 supercomputer was announced and marveling that it didn’t even use virtual memory. It was believed at the time that only real-mode memory access could deliver the performance needed.

We have come a long way in the nearly 40 years since the Cray 1 was announced. This #42 result was run not just using virtual memory but with virtual memory in a guest operating system running under a hypervisor. This is the only fully virtualized, multi-tenant super computer on the Top500 and it shows what is possible as the virtualization tax continues to fall. This is an awesome result and many more virtualization improvements are coming over the next 2 to 3 years.

Commodity Networks can Compete at the Top of the Performance Spectrum:

This is the only Top500 entrant below number 128 on the list that is not running either Infiniband or a proprietary, purpose-built network. This result at #42 is an all Ethernet network showing that a commodity network, if done right, can produce industry leading performance numbers.

What’s the secret? 10Gbps directly the host is the first part. The second is full non-blocking networking fabric where all systems can communicate at full line rate at the same time. Worded differently, the network is not oversubscribed. See Datacenter Networks are in my Way for more on the problems with existing datacenter networks.

Commodity Ethernet networks continue to borrow more and more implementation approaches and good network architecture ideas from Infiniband, scale economics continues to drive down costs so non-blocking networks are now practical and affordable, and scale economics are pushing rapid innovation. Commodity equipment in a well-engineered overall service is where I see the future of networking continuing to head.

Anyone can own a Supercomputer for an hour:

You can’t rent supercomputing time by the hour from Lawrence Livermore National Laboratory. Sandia is not doing it either. But you can have a top50 supercomputer for under $2,600/hour. That is one of the world’s most powerful high performance computing systems with 1,064 nodes and 8,512 cores for under $3k/hour. For those of you not needing quite this much power at one time, that’s $0.05/core hour which is ½ of the previous Amazon Web Services HPC system cost.

Single node speeds and feeds:

· Processors: 8-core, 2 socket Intel Xeon @ 2.6 Ghz with hyperthreading

· Memory: 60.5GB

· Storage: 3.37TB direct attached and Elastic Block Store for remote storage

· Networking: 10Gbps Ethernet with full bisection bandwidth within the placement group

· Virtualized: Hardware Assisted Virtualization

· API: cc2.8xlarge

Overall Top500 Result:

· 1064 nodes of cc2.8xlarge

· 240.09 TFlops at an excellent 67.8% efficiency

· $2.40/node hour on demand

· 10Gbps non-blocking Ethernet networking fabric

Database Intensive Computing:

This is a database machine masquerading as a supercomputer. You don’t have to use the floating point units to get full value from renting time on this cluster. It’s absolutely a screamer as an HPC system. But it also has the potential to be the world’s highest performing MapReduce system (Elastic Map Reduce) with a full bisection bandwidth 10Gbps network directly to each node. Any database or general data intensive workload with high per-node computational costs and/or high inter-node traffic will run well on this new instance type.

If you are network bound, compute bound, or both, the EC2 cc2.8xlarge instance type could be the right answer. And, the amazing thing is that the cc2 instance type is ½ the cost per core of the cc1 instance.

Supercomputing is now available to anyone for $0.05/core hour. Go to http://aws.amazon.com/hpc-applications/ and give it a try. You no longer need to be a national lab or a government agency to be able run one of the biggest supercomputers in the world.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Storage Infrastructure Behind Facebook Messages

James Hamilton — Tue, 25 Oct 2011 21:03:10 GMT

One of the talks that I particularly enjoyed yesterday at HPTS 2011 was Storage Infrastructure Behind Facebook Messages by Kannan Muthukkaruppan. In this talk, Kannan talked about the Facebook store for chats, email, SMS, & messages.

This high scale storage system is based upon HBase and Haystack. HBase is a non-relational, distributed database very similar to Google’s Big Table. Haystack is simple file system designed by Facebook for efficient photo storage and delivery. More on Haystack at: Facebook Needle in a Haystack.

In this Facebook Message store, Haystack is used to store attachments and large messages. HBase is used for message metadata, search indexes, and small messages (avoiding the second I/O to Haystack for small messages like most SMS).

Facebook Messages takes 6B+ messages a day. Summarizing HBase traffic:

· 75B+ R+W ops/day with 1.5M ops/sec at peak

· The average write operation inserts 16 records across multiple column families

· 2PB+ of cooked online data in HBase. Over 6PB including replication but not backups

· All data is LZO compressed

· Growing at 250TB/month

The Facebook Messages project timeline:

· 2009/12: Project started

· 2010/11: Initial rollout began

· 2011/07: Rollout completed with 1B+ accounts migrated to new store

· Production changes:

o 2 schema changes

o Upgraded to Hfile 2.0

They implemented a very nice approach to testing where, prior to release, they shadowed the production workload to the test servers.

After going into production the continued the practice of shadowing the real production workload into the test cluster to test before going into production:

The list of scares and scars from Kannan:

· Not without our share of scares and incidents:

o s/w bugs. (e.g., deadlocks, incompatible LZO used for bulk imported data, etc.)

§ found a edge case bug in log recovery as recently as last week!

· performance spikes every 6 hours (even off-peak)!

o cleanup of HDFS’s Recycle bin was sub-optimal! Needed code and config fix.

· transient rack switch failures

· Zookeeper leader election took than 10 minutes when one member of the quorum died. Fixed in more recent version of ZK.

· HDFS Namenode – SPOF

· flapping servers (repeated failures)

· Sometimes, tried things which hadn’t been tested in dark launch!

o Added a rack of servers to help with performance issue

§ Pegged top of the rack network bandwidth!

§ Had to add the servers at much slower pace. Very manual .

§ Intelligent load balancing needed to make this more automated.

· A high % of issues caught in shadow/stress testing

· Lots of alerting mechanisms in place to detect failures cases

o Automate recovery for a lots of common ones

o Treat alerts on shadow cluster as hi-pri too!

· Sharding service across multiple HBase cells also paid off

Kannan’s slides are posted at: http://mvdirona.com/jrh/TalksAndPapers/KannanMuthukkaruppan_StorageInfraBehindMessages.pdf

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Microsoft COSMOS at HPTS

James Hamilton — Tue, 25 Oct 2011 16:37:20 GMT

Rough notes from a talk on COSMOS, Microsoft’s internal Map reduce systems from HPTS 2011. This is the service Microsoft uses internally to run MapReduce jobs. Interesting, Microsoft plans to use Hadoop in the external Azure service even though COSMOS looks quite good: Microsoft Announces Open Source Based Cloud Service. Rough notes below:

Talk: COSMOS: Big Data and Big Challenges

Speaker: Ed Harris

· Petabyte storage and computation systems

· Used primarily by search and advertising inside Microsoft

· Operated as a service with just over 4 9s of availability

· Massively parallel processing based upon Dryad

o Dryad is very similar to MapReduce

· Use SCOPE (structured Computation Optimized for Parallel Execution) over Dryad

o A SQL-like language with an optimizers implemented over Dryad

· They run hundreds of virtual clusters. In this model, internal Microsoft teams buy servers and given them to COSMOS and are subsequently assured at least these resources

o Average 85% CPU over the cluster

· Ingest 1 to 2 PB/day

· Roughly 30% of the Search fleet is running COSMOS

· Architecture:

o Store Layer

§ Many extent nodes store and compress streams

§ Streams are sequences of extents

§ CSM: Cosmos Store Layer handles names, streams, and replication

· First level compression is light. Data that is kept more than a week is more aggressively compressed after a week on the assumption that data that lives a week will likely live longer

o Execution Layer:

§ Jobs queue up on virtual clusters and then executed

o SCOPE Layer

§ Compiler and optimizer for SCOPE

§ Ed said that the optimizer is a branch of the SQL Server optimizer

· They have 60+ Phd internships each year and hire ~30 a year

James Hamilton

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

EMCs Joe Tucci on the Cloud and Big Data

James Hamilton — Thu, 20 Oct 2011 14:24:29 GMT

Last night EMC Chief Executive Joe Tucci laid out his view of where the information processing world is going over the next decade and where EMC will focus. His primary point was cloud computing is the future and big data is the killer app for the cloud. He laid out the history of big transitions in our industry and argued the big discontinuities were always driven by a killer application. He sees the cloud as the next big and important transition for our industry.

This talk was presented as part of the University of Washington Distinguished Lecturer Series. With six TV cameras covering the action, there were nearly as many as some University of Washington Huskies games and it was well attended. The next talk in the series will be Bill Gates on October 27 presenting The Opportunity Ahead: A Conversation with Bill Gates. I’ll be presenting Internet Scale Storage on November 1^st.

If you are interested in any of the talks in the series, all are open to the public and the upcoming schedule is posted at: http://www.cs.washington.edu/news/newdlshome.html.

The most notable statistic from the Joe Tucci talk was the massive investment that EMC is making mergers and acquisitions. He said over the next 5 years, EMC will spend $10.5B in R&D – this number alone is amazingly large -- but what I found really startling was they expect to spend even more purchasing companies. They expect to spend $14.0B on M&A during this same period. That’s nearly $3B/year from just a single company. Amazing.

With many large companies increasingly looking to the startup community for new ideas and innovation, there is incredible opportunity for startups. Joe emphasized the opportunity, saying that Washington in general and especially the University of Washington will likely be the source of many of these new companies. As large companies lean more on the startup community for new ideas, products, and services, it’s a good time to be starting a company.

My rough notes from the talk:

· IDC reports:

o This decade WW information content will grow 44x (0.9 zettabytes to 35.2)

o 90% unstructured

· Big data has arrived

o Mobile sensors

o Social media

o Video surveillance

o Smart grids

o Gene sequencing

o Medical imaging

o Geophysical exploration

· 73% maintaining existing infrastructure (true for 10 years)

o JRH: I’ve heard this statistic before but it seems like nearly has to be the case the most companies are spending at least 3/4s of their investment continuing to running the business and around a ¼ on new applications. The statistic is usually presented as a problem but it feels like it might be close to the right ratio.

· 3D movie is about a petabyte with all camera angles and footage included

· The average company is attacked 300 times per week

o All CIO say this is way light – my home router gets nailed that many times in a good hour

· IT staffing will increase less than 50% in next 10 years but the data under management will grow faster.

o JRH: Again this seems like the desirable outcome where the data under management should be able to grow far faster than administrative team

· EMCs Mission: To lead customers towards a hybrid cloud

o Leading customers to x86 based private clouds and hybrid clouds

o Burst, test & development, etc. into the public cloud

o Hybrid cloud between private and public is the “big winner”

· VM is basically a cloud operating systems

o EMC still owns 80% of VMWare

o There are now more than virtual machines shipped than physical machines

o 62% virtualized out of the gate

· Applications like SAP, Oracle, and Microsoft are now available in the cloud

· Killer app for the cloud is big data

o Real time data analytics

· New end user computing

o IOS devices, android, windows, …

· Tenets of cloud computing

o Efficiency, control, choice => Agility

o Control through policy, service levels, and cost

· Big competitors

o IBM, HP, Cisco, Microsoft, …

o EMC is big at $20B but not close to as big as these incumbents

o JRH: I’ve never thought of EMC as the small, nimble competitor but I guess it’s all relative

· Recent acquisitions in drive to cloud & big data

o Isilon

o Greenplum

o Datadomain

o RSA

· Mammoth 5 year M&A plan: roughly ½ of investments in R&D and ½ in M&A

o 14.0B M&A

o 10.5B: R&D

· EMC has 14,000 sales people so there is huge potential synergy in any acquisition

o Adding a 14,000 person sales team to any reasonable product is going to produce considerable new revenue quickly

· EMC is now 152 in fortune 500

o Revenue is $17B

o Free cash flow: $3.4b

Thanks to Ed Lazowska for hosting this talk and many in the University of Washington Distinguished Lecturer Series.

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Microsoft Announces Open Source based Cloud Service

James Hamilton — Thu, 13 Oct 2011 15:08:10 GMT

We see press releases go by all the time and most of them deserve the yawn they get. But, one caught my interest yesterday. At the PASS Summit conference Microsoft Vice President Ted Kummert announced that Microsoft will be offering a big data solution based upon Hadoop as part of SQL Azure. From the Microsoft press release, “Kummert also announced new investments to help customers manage big data, including an Apache Hadoop-based distribution for Windows Server and Windows Azure and a strategic partnership with Hortonworks Inc.”

Clearly this is a major win for the early startup Hortonworks. Hortonworks is a spin out of Yahoo! and includes many of the core contributors to the Apache Hadoop distribution: Hortonwoks Taking Hadoop to Next Level.

This announcement is also a big win for the MapReduce processing model. First invented at Google and published in MapReduce: Simplified Data Processing on Large Clusters. The Apache Hadoop distribution is an open source implementation of MapReduce. Hadoop is incredibly widely used with Yahoo! running more than 40,000 nodes of Hadoop with their biggest single cluster now at 4,500 servers. Facebook runs a 1,100 node cluster and a second 300 node cluster. Linked in runs many clusters including deployments of 1,200, 580, and 120 nodes. See the Hadoop Powered By Page for many more examples.

In the cloud, AWS began offering Elastic MapReduce back in early 2009 and has been expanding the features supported by this offering steadily over the last couple of years adding support for Reserved Instances, Spot Instances, and Cluster Compute instances (on a 10Gb non-oversubscribed network – MapReduces just loves high bandwidth inter-node connectivity)and support for more regions with EMR available in Northern Virginia, Northern California, Ireland, Singapore, and Tokyo.

Microsoft expects to have a pre-production (what they refer to as a "community technology Preview") version of a Hadoop service available by the “end of 2011”. This is interesting for a variety of reasons. First, its more evidence of the broad acceptance and applicability of the MapReduce model. What is even more surprising is that Microsoft has decided in this case to base their MapReduce offering upon open source Hadoop rather than the Microsoft internally developed MapReduce service called Cosmos which is used heavily by the Bing search and advertising teams. The What is Dryad blog entry provides a good description of Cosmos and some of the infrastructure build upon the Cosmos core including Dryad, DryadLINQ, and SCOPE.

As surprising as it is to see Microsoft planning to offer MapReduce based upon open source rather than upon the internally developed and heavily used Cosmos platform, it’s even more surprising that they hope to contribute changes back to the open source community saying “Microsoft will work closely with the Hadoop community and propose contributions back to the Apache Software Foundation and the Hadoop project.”

· Microsoft Press Release: Microsoft Expands Data Platform

· Hortonsworks Press Release: Hortonworks to Extend Apache Hadoop to Windows Users

· Hortonworks Blog Entry: Bringing Apache Hadoop to Windows

Past MapReduce postings on Perspectives:

· MapReduce in CACM

· MapReduce: A Minor Step Forward

· Hadoop Summit 2010

· Hadoop Summit 2008

· Hadoop Wins TeraSort

· Google MapReduce Wins TeraSort

· HadoopDB: MapReduce over Relational Data

· Hortonworks Taking Hadoop to Next Level

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

2011 European Data Center Summit

James Hamilton — Wed, 25 May 2011 13:52:18 GMT

The European Data Center Summit 2011 was held yesterday at SihlCity CinCenter in Zurich. Google Senior VP Urs Hoelzle kicked off the event talking about why data center efficiency was important both economically and socially. He went on to point out that the oft quoted number that US data centers represent is 2% of total energy consumption is usually mis-understood. The actual data point is that 2% of the US energy budget is spent on IT of which the vast majority is client side systems. This is unsurprising but a super important clarification. The full breakdown of this data:

· 2% of US power

o Datacenters: 14%

o Telecom: 37%

o Client Device: 50%

The net is that 14% of 2% or 0.28% of the US power budget is consumed in datacenters. This is a far smaller but still a very relevant number. In fact, that is the primary motivator behind the conference: how to get the best practices from industry leaders in datacenter efficiency available more broadly .

To help understand why this is important,

· Of the 0.28% energy consumption by datacenters:

o Small: 41%

o Medium: 31%

o Large: 28%

This later set of statistics predictably shows that the very largest data centers consume 28% of the data center energy budget while small and medium centers consume 72%. High scale datacenter operates have large staffs of experts focused on increasing energy efficiency but small and medium sized centers can’t afford this overhead at their scale. Urs’s point and the motivation behind the conference is we need to get industry best practices available to all data center operations.

The driving goal behind the conference is that extremely efficient datacenter operations are possible using only broadly understood techniques. No magic is required. It is true that the very large operators will continue to enjoy even better efficiency but existing industry best practices can easily get even small operators with limited budgets to within a few points of the same efficiency levels.

Using Power Usage Effectiveness as the measure while the industry leaders are at 1.1 to 1.2 where 1.2 means that every watt delivered to the servers requires 1.2 watts to be deliverd from the utility. Effectively it is a measure of the overhead or efficiency of the datacenter infrastructure. Unfortunately the average remains in the 1.8 to 2.0 range and the worst facilities can be as poor as 3.0.

Summarizing: Datacenters consume 0.28% of the annual US energy budget. 72% of these centers are small and medium sized centers that tend towards the lower efficiency levels.

The Datacenter Efficiency conference focused on making cost effective techniques more broadly understood showing how a PUE of 1.5 is available to all without large teams of experts or huge expense. This is good for the environment and less expensive to operate.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Software Load Balancing using Software Defined Networking

James Hamilton — Fri, 20 May 2011 18:55:51 GMT

I invited Nikhil Handigol to present at Amazon earlier this week. Nikhil is a Phd candidate at Stanford University working with networking legend Nick McKeown on the Software Defined Networking team. Software defined networking is an concept coined by Nick where the research team is separating the networking control plane from the data plane. The goal is a fast and dumb routing engine with the control plane factored out and supporting an open programming platform.

From Nikil’s presentation, we see the control plane hoisted up to a central, replicated network O/S configuring the distributed routing engines in each switch.

One implementation of software defined networking is OpenFlow where each router supports the OpenFlow protocol and a central OpenFlow Controller computes routing tables that are installed in each router:

What makes OpenFlow especially interesting is that it’s simple, easy to implement, and getting broad industry support with the Open Networking Foundation as the central organizing body. The Open Networking Foundation’s primary mission is to advance software defined networking using OpenFlow as the protocol. Founding members of the Open Networking Foundation are Deutsche Telekom, Facebook, Google, Microsoft, Verizon, and Yahoo!. Also included are networking equipment providers including: Broadcom, Dell, Cisco, Force10, HP, Juniper, Marvell, Mellanox, and many others.

Today, most networking equipment is shipped as a vertically integrated stack including both the control and data planes. There are many reasons why this is not good for the industry. The Stanford team argues it blocks innovation in that researches can’t try new protocols with a closed stack without a programming model. I agree. This is a problem for both academia and industry but my dislike of the current model is much broader. In Networking: The Last Bastion of Mainframe Computing, I made the case that this vertically integrated approach is artificially holding prices high and slowing the pace of innovation. A quick summary of the argument:

When networking equipment is purchased, it’s packaged as a single sourced, vertically integrated stack. In contrast, in the commodity server world, starting at the most basic component, CPUs are multi-sourced. We can get CPUs from AMD and Intel. Compatible servers built from either Intel or AMD CPUs are available from HP, Dell, IBM, SGI, ZT Systems, Silicon Mechanics, and many others. Any of these servers can support both proprietary and open source operating systems. The commodity server world is open and multi-sourced at every layer in the stack.

Open, multi-layer hardware and software stacks encourage innovation and rapidly drive down costs. The server world is clear evidence of what is possible when such an ecosystem emerges.

I’m excited about software defined networking because it provides a clean interface allowing switch providers to both innovate and compete. An additional benefit is that SDN allows innovation and experimentation at the network protocol layer.

In Nikil’s talk last week at Amazon, he explored integrating load balancing functionality into the network routing fabric. The team started with the hypothesis that load balancing is really just smart routing. They then implemented a distributed load balancing fabric by adding load balancing support to network routers using Software Defined Networking. Essentially they distribute the load balancing functionality throughout the network. What’s unusual here is that the ideas could be tested and tried over a 9 campus, North American wide network with only 500 lines of code. With conventional network protocol stacks, this research work would have been impossible in that vendors don’t open up protocol stacks. And, even if they did, it would have been complex and very time consuming.

Video Demo of the distributed load balancer: http://www.youtube.com/watch?v=Sfqofxdk1gE
Nikil’s slides from his Amazon Talk: http://www.mvdirona.com/jrh/TalksAndPapers/Handigol-Amazon-May2011.pdf

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Prioritizing Principlas in "On Designing and Deploying Internet-Scale Services"

James Hamilton — Sat, 26 Mar 2011 16:21:16 GMT

Brad Porter is Director and Senior Principal engineer at Amazon. We work in different parts of the company but I have known him for years and he’s actually one of the reasons I ended up joining Amazon Web Services. Last week Brad sent me the guest blog post that follows where, on the basis of his operational experience, he prioritizes the most important points in the Lisa paper On Designing and Deploying Internet-Scale Services.

--jrh

Prioritizing the Principles in "On Designing and Deploying Internet-Scale Services"

By Brad Porter

James published what I consider to be the single best paper to come out of the highly-available systems world in many years. He gave simple practical advice for delivering on the promise of high-availability. James presented “On Designing and Deploying Internet-Scale Services” at Lisa 2007.

A few folks have commented to me that implementing all of these principles is a tall hill to climb. I thought I might help by highlighting what I consider to be the most important elements and why.

1. Keep it simple

Much of the work in recovery-oriented computing has been driven by the observation that human errors are the number one cause of failure in large-scale systems. However, in my experience complexity is the number one cause of human error.

Complexity originates from a number of sources: lack of a clear architectural model, variance introduced by forking or branching software or configuration, and implementation cruft never cleaned up. I'm going to add three new sub-principles to this.

Have Well Defined Architectural Roles and Responsibilities: Robust systems are often described as having "good bones." The structural skeleton upon which the system has evolved and grown is solid. Good architecture starts from having a clear and widely shared understanding of the roles and responsibilities in the system. It should be possible to introduce the basic architecture to someone new in just a few minutes on a white-board.

Minimize Variance: Variance arises most often when engineering or operations teams use partitioning typically through branching or forking as a way to handle different use cases or requirements sets. Every new use case creates a slightly different variant. Variations occur along software boundaries, configuration boundaries, or hardware boundaries. To the extent possible, systems should be architected, deployed, and managed to minimize variance in the production environment.

Clean-Up Cruft: Cruft can be defined as those things that clearly should be fixed, but no one has bothered to fix. This can include unnecessary configuration values and variables, unnecessary log messages, test instances, unnecessary code branches, and low priority "bugs" that no one has fixed. Cleaning up cruft is a constant task, but it is a necessary to minimize complexity.

2. Expect failures

At its simplest, a production host or service need only exist in one of two states: on or off. On or off can be defined by whether that service is accepting requests or not. To "expect failures" is to recognize that "off" is always a valid state. A host or component may switch to the "off" state at any time without warning.

If you're willing to turn a component off at any time, you're immediately liberated. Most operational tasks become significantly simpler. You can perform upgrades when the component is off. In the event of any anomalous behavior, you can turn the component off.

3. Support version roll-back

Roll-back is similarly liberating. Many system problems are introduced on change-boundaries. If you can roll changes back quickly, you can minimize the impact of any change-induced problem. The perceived risk and cost of a change decreases dramatically when roll-back is enabled, immediately allowing for more rapid innovation and evolution, especially when combined with the next point.

4. Maintain forward-and-backward compatibility

Forcing simultaneous upgrade of many components introduces complexity, makes roll-back more difficult, and in some cases just isn't possible as customers may be unable or unwilling to upgrade at the same time.

If you have forward-and-backwards compatibility for each component, you can upgrade that component transparently. Dependent services need not know that the new version has been deployed. This allows staged or incremental roll-out. This also allows a subset of machines in the system to be upgraded and receive real production traffic as a last phase of the test cycle simultaneously with older versions of the component.

5. Give enough information to diagnose

Once you have the big ticket bugs out of the system, the persistent bugs will only happen one in a million times or even less frequently. These problems are almost impossible to reproduce cost effectively. With sufficient production data, you can perform forensic diagnosis of the issue. Without it, you're blind.

Maintaining production trace data is expensive, but ultimately less expensive than trying to build the infrastructure and tools to reproduce a one-in-a-million bug and it gives you the tools to answer exactly what happened quickly rather than guessing based on the results of a multi-day or multi-week simulation.

I rank these five as the most important because they liberate you to continue to evolve the system as time and resource permit to address the other dimensions the paper describes. If you fail to do the first five, you'll be endlessly fighting operational overhead costs as you attempt to make forward progress.

If you haven't kept it simple, then you'll spend much of your time dealing with system dependencies, arguing over roles & responsibilities, managing variants, or sleuthing through data/config or code that is difficult to follow.
If you haven't expected failures, then you'll be reacting when the system does fail. You may also be dealing with complicated change-management processes designed to keep the system up and running while you're attempting to change it.
If you haven't implemented roll-back, then you'll live in fear of your next upgrade. After one or two failures, you will hesitate to make any further system change, no matter how beneficial.
Without forward-and-backward compatibility, you'll spend much of your time trying to force dependent customers through migrations.
Without enough information to diagnose, you'll spend substantial amounts of time debugging or attempting to reproduce difficult-to-find bugs.

I'll end with another encouragement to read the paper if you haven't already: "On Designing and Deploying Internet-Scale Services"

From Perspectives.

Yahoo! Compute Coop Design

James Hamilton — Sat, 05 Mar 2011 21:58:04 GMT

Chris Page, Director of Climate & Energy Strategy at Yahoo! spoke at the 2010 Data Center Efficiency Summit and presented Yahoo! Compute Coop Design.

The primary attributes of the Yahoo! design are: 1) 100% free air cooling (no chillers), 2) slab concrete floor, 3) use of wind power to augment air handling units, and 4) pre-engineered building for construction speed.

Chris reports the idea to orient the building such that the wind force on the external wall facing the dominant wind direction and use this higher pressure to assist the air handling units was taken from looking at farm buildings in the Buffalo, New York area. An example given was the use of natural cooling in chicken coops.

Chiller free data centers are getting more common (Chillerless Data Center at 95F) but the design approach is still far from common place. The Yahoo! team reports they run the facility at 75F and use evaporative cooling to attempt to hold the server approach temperatures down below 80F using evaporative cooling. A common approach and the one used by Microsoft in the chillerless datacenter at 95F example is install low efficiency coolers for those rare days when they are needed on the logic that low efficiency is cheap and really fine for very rare use. The Yahoo! approach is to avoid the capital cost and power consumption of chillers entirely by allowing the cold aisle temperatures to rise to 85F to 90F when they are unable to hold the temperature lower. They calculate they will only do this 34 hours a year which is less than 0.4% of the year.

Reported results:

· PUE at 1.08 with evaporative cooling and believe they could do better in colder climates

· Save ~36 million gallons of water / yr, compared to previous water cooled chiller plant w/o airside economization design

· Saves over ~8 million gallons of sewer discharge / yr (zero discharge design)

· Lower construction cost (not quantified)

· 6 months from dirt to operating facility

Chris’ slides: http://dcee.svlg.org/images/stories/pdf/DCES10/airandwatercafe130.pdf (the middle slides of the 3 slide deck set)

Thanks to Vijay Rao for sending this my way.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Dileep Bhandarkar on Datacenter Energy Efficiency

James Hamilton — Sun, 20 Feb 2011 15:00:07 GMT

Dileep Bhandarkar presented the keynote at Server design Summit last December. I can never find the time to attend trade shows so I often end up reading slides instead. This one had lots of interesting tidbits so I’m posting a pointer to the talk and my rough notes here.

Dileep’s talk: Watt Matters in Energy Efficiency

My notes:

· Microsoft Datacenter Capacities:

o Quincy WA: 550k sq ft, 27MW

o San Antonio Tx: 477k sq ft, 27MW

o Chicago, Il: 707k sq ft, 60MW

§ Containers on bottom floor with “medium reliability” (no generators) and standard rooms on top floor with full power redundancy

o Dublin, Ireland: 570k sq ft, 27MW

· Speeds and feeds from Microsoft Consumer Cloud Services:

o Windows Live: 500M IDs

o Live Hotmail: 355M Active Accounts

o Live Messenger: 303M users

o Bing: 4B Queries/month

o Xbox Live: 25M users

o adCenter: 14B Ads served/month

o Exchange Hosted Services: 2 to 4B emails/day

· Datacenter Construction Costs

o Land: <2%

o Shell: 5 to 9%

o Architectural: 4 to 7%

o Mechanical & Electrical: 70 to 85%

· Summarizing the above list, we get 80% of the costs scaling with power consumption and 10 to 20% scaling with floor space. Reflect on that number and you’ll understand why I think the industry is nuts to be focusing on density. See Why Blade Servers Aren’t the Answer to All Questions for more detail on this point – I think it’s a particularly important one.

· Reports that datacenter build costs are $10M to $20M per MW and server TCO is the biggest single category. I would use the low end of this spectrum for a cost estimator with inexpensive facilities in the $9M to $10M/MW range. See Overall Datacenter Costs.

· PUE is a good metric for evaluating datacenter infrastructure efficiency but Microsoft uses best server performance per watt per TCO$

o Optimize the server design and datacenter together

· Cost-Reduction Strategies:

o Server cost reduction:

§ Right size server: Low Power Processors often offer best performance/watt (see Very Low-Power Server Progress for more)

§ Eliminate unnecessary components (very small gain)

§ Use higher efficiency parts

§ Optimize for server performance/watt/$ (cheap and low power tend to win at scale)

o Infrastructure cost reduction:

§ Operate at higher temperatures

§ Use free air cooling & Eliminate chillers (More detail at: Chillerless Datacenter at 95F)

§ Use advanced power management with power capping to support power over-subscription with peak protection (see the 2007 paperhttp://research.google.com/pubs/pub32980.html for the early work on this topic)

· Custom Server Design:

o 2 socket, half-width server design (6.3”W x 16.7”L)

o 4x SATA HDD connectors

o 4x DIMM slots per CPU socket

o 2x 1GigE NIC

o 1x 16x PCIe slot

· Custom Rack Design:

o 480 VAC 3p power directly to the rack (higher voltage over a given conductor size reduces losses in distribution)

o Very tall 56 RU rack (over 98” overall height)

o 12VDC distribution within the rack from two combined power supplies with distributed UPS

o Power Supplies (PSU)

§ Input is 480VAC 3p

§ Output: 12V DC

§ Servers are 12VDC only boards

§ Each PSU is 4.5KW

§ 2 PSUs/rack so rack is 9.0KW max

o Distributed UPS

§ Each PSU includes an UPS made up of 4 groups of 4013.2V batteries

§ Overall 160 discrete batteries per UPS

§ Technology not specified but based upon form factor and rough power estimate, I suspect they may be Li-ion 18650. See Commodity Parts for a note on these units.

o By putting 2 PSUs per rack they avoid transporting low voltage (12VDC) further than 1/3 of a rack distance (under 1 yard) and only 4.5 KW is transported so moderately sized bus bars can be used.

o Rack Networking:

§ Rack networking is interesting with 2 to 4 tor switches per rack. We know the servers are 1 GigE connected and there are up to 96 per rack which yields two possibilities: 1) they are using 24 port GigE switches or 2) they are using 48 port GigE and fat tree network topology (see VL2: A Scalable and Flexible Datacenter Network for more on a fat tree). 24 port TORs are not particularly cost effective, so I would assume 48x1GigE TORs in a fat tree design which is nice work.

o Report that the CPUs can be lower power parts including Intel Atom, AMD Bobcat, and ARM (more on lower powered processors in server applications: http://perspectives.mvdirona.com/2011/01/16/NVIDIAProjectDenverARMPoweredServers.aspx)

· Power proportionality: Shows that server at 0% load consumes 32% of peak power (this is an amazingly good server – 45% to 60% is much closer to the norm)

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Google Megastore: The Data Engine Behind GAE

James Hamilton — Sun, 09 Jan 2011 18:38:39 GMT

Megastore is the data engine supporting the Google Application Engine. It’s a scalable structured data store providing full ACID semantics within partitions but lower consistency guarantees across partitions.

I wrote up some notes on it back in 2008 Under the Covers of the App Engine Datastore and posted Phil Bernstein’s excellent notes from a 2008 SIGMOD talk: Google Megastore. But there has been remarkably little written about this datastore over the intervening couple of years until this year’s CIDR conference papers were posted. CIDR 2011 includes Megastore: Providing Scalable, Highly Available Storage for Interactive Services.

My rough notes from the paper:

· Megastore is built upon BigTable

· Bigtable supports fault-tolerant storage within a single datacenter

· Synchronous replication based upon Paxos and optimized for long distance inter-datacenter links

· Partitioned into a vast space of small databases each with its own replicated log

· Each log stored across a Paxos cluster

· Because they are so aggressively partitioned, each Paxos group only has to accept logs for operations on a small partition. However, the design does serialize updates on each partition

· 3 billion writes and 20 billion read transactions per day

· Support for consistency unusual for a NoSQL database but driven by (what I believe to be) the correct belief that inconsistent updates make many applications difficult to write (see I Love Eventual Consistency but …)

· Data Model:

· The data model is declared in a strongly typed schema

· There are potentially many tables per schema

· There are potentially many entities per table

· There are potentially many strongly typed properties per entity

· Repeating properties are allowed

· Tables can be arranged hierarchically where child tables point to root tables

· Megastore tables are either entity group root tables or child tables

· The root table and all child tables are stored in the same entity group

· Secondary indexes are supported

· Local secondary indexes index a specific entity group and are maintained consistently

· Global secondary indexes index across entity groups are asynchronously updates and eventually consistent

· Repeated indexes: supports indexing repeated values (e.g. photo tags)

· Inline indexes provide a way to denormalize data from source entities into a related target entity as a virtual repeated column.

· There are physical storage hints:

· “IN TABLE” directs Megastore to store two tables in the same underlying BigTable

· “SCATTER” attribute prepends a 2 byte hash to each key to cool hot spots on tables with monotonically increasing values like dates (e.g. a history table).

· “STORING” clause on an index supports index-only-access by redundantly storing additional data in an index. This avoids the double access often required of doing a secondary index lookup to find the correct entity and then selecting the correct properties from that entity through a second table access. By pulling values up into the secondary index, the base table doesn’t need to be accessed to obtain these properties.

· 3 levels of read consistency:

· Current: Last committed value

· Snapshot: Value as of start of the read transaction

· Inconsistent reads: used for cross entity group reads

· Update path:

· Transaction writes its mutations to the entity groups write-ahead log and then apply the mutations to the data (write ahead logging).

· Write transaction always begins with a current read to determine the next available log position. The commit operation gathers mutations into a log entry, assigns an increasing timestamp, and appends to log which is maintained using paxos.

· Update rates within a entity group are seriously limited by:

· When there is log contention, one wins and the rest fail and must be retried.

· Paxos only accepts a very limited update rate (order 10^2 updates per second).

· Paper reports that “limiting updates within an entity group to a few writes per second per entity group yields insignificant write conflicts”

· Implication: programmers must shard aggressively to get even moderate update rates and consistent update across shards is only supported using two phase commit which is not recommended.

· Cross entity group updates are supported by:

· Two-phase commit with the fragility that it brings

· Queueing and asynchronously applying the changes

· Excellent support for backup and redundancy:

· Synchronous replication to protect against media failure

· Snapshots and incremental log backups

Overall, an excellent paper with lots of detail on a nicely executed storage system. Supporting consistent read and full ACID update semantics is impressive although the limitation of not being able to update an entity group at more than a “few per second” is limiting.

The paper: http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf.

Thanks to Zhu Han, Reto Kramer, and Chris Newcombe for all sending this paper my way.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Amazon Route 53 DNS Service

James Hamilton — Mon, 06 Dec 2010 13:37:15 GMT

Even working in Amazon Web Services, I’m finding the frequency of new product announcements and updates a bit dizzying. It’s amazing how fast the cloud is taking shape and the feature set is filling out. Utility computing has really been on fire over the last 9 months. I’ve never seen an entire new industry created and come fully to life this fast. Fun times.

Before joining AWS, I used to say that I had an inside line on what AWS was working upon and what new features were coming in the near future. My trick? I went to AWS customer meetings and just listened. AWS delivers what customers are asking for with such regularity that it’s really not all that hard to predict new product features soon to be delivered. This trend continues with today’s announcement. Customers have been asking for a Domain Name Service with consistency and, today, AWS is announcing the availability of Route 53, a scalable, highly-redundant and reliable, global DNS service.

The Domain Name System is essentially a global, distributed database that allows various pieces of information to be associated with a domain name. In the most common case, DNS is used to look up the numeric IP address for an domain name. So, for example, I just looked up Amazon.com and found that one of the addresses being used to host Amazon.com is 207.171.166.252. And, when your browser accessed this blog (assuming you came here directly rather than using RSS) it would have looked up perspectives.mvdirona.com to get an IP address. This mapping is stored in an DNS “A” (address) record. Other popular DNS records are CNAME (canonical name), MX (mail exchange), and SPF (Sender Policy Framework). A full list of DNS record types is at: http://en.wikipedia.org/wiki/List_of_DNS_record_types. Route 53 currently supports:

• A (address record)

• AAAA (IPv6 address record)

• CNAME (canonical name record)

• MX (mail exchange record)

• NS (name server record)

• PTR (pointer record)

• SOA (start of authority record)

• SPF (sender policy framework)

• SRV (service locator)

• TXT (text record)

DNS, on the surface, is fairly simple and is easy to understand. What is difficult with DNS is providing absolute rock-solid stability at scales ranging from a request per day on some domains to billions on others. Running DNS rock-solid, low-latency, and highly reliable is hard. And it’s just the kind of problem that loves scale. Scale allows more investment in the underlying service and supports a wide, many-datacenter footprint.

The AWS Route 53 Service is hosted in a global network of edge locations including the following 16 facilities:

· United States

• Ashburn, VA

• Dallas/Fort Worth, TX

• Los Angeles, CA

• Miami, FL

• New York, NY

• Newark, NJ

• Palo Alto, CA

• Seattle, WA

• St. Louis, MO

· Europe

• Amsterdam

• Dublin

• Frankfurt

• London

· Asia

• Hong Kong

• Tokyo

• Singapore

Many DNS lookups are resolved in local caches but, when there is a cache miss, it will need to be routed back to the authoritative name server. The right approach to answering these requests with low latency is to route to the nearest datacenter hosting an appropriate DNS server. In Route 53 this is done using anycast. Anycast is a cool routing trick where the same IP address range is advertised to be at many different locations. Using this technique, the same IP address range is advertized as being in each of the world-wide fleet of datacenters. This results in the request being routed to the nearest facility from a network perspective.

Route 53 routes to the nearest datacenter to deliver low-latency, reliable results. This is good but Route 53 is not the only DNS service that is well implemented over a globally distributed fleet of datacenters. What makes Route 53 unique is it’s a cloud service. Cloud means the price is advertised rather than negotiated. Cloud means you make an API call rather than talking to a sales representative. Cloud means it’s a simple API and you don’t need professional services or a customer support contact. And cloud means its running NOW rather than tomorrow morning when the administration team comes in. Offering a rock-solid service is half the battle but it’s the cloud aspects of Route 53 that are most interesting.

Route 53 pricing is advertised and available to all:

· Hosted Zones: $1 per hosted zone per month

· Requests: $0.50 per million queries for first billion queries and $0.25 per million queries over 1B month

You can have it running in less time than it took to read this posting. Go to: ROUTE 53 Details. You don’t need to talk to anyone, negotiate a volume discount, hire a professional service team, call the customer support group, or wait until tomorrow. Make the API calls to set it up and, on average, 60 seconds later you are fully operating.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Availability in Globally Distributed Storage Systems

James Hamilton — Mon, 29 Nov 2010 13:55:51 GMT

I love high-scale systems and, more than anything, I love data from real systems. I’ve learned over the years that no environment is crueler, less forgiving, or harder to satisfy than real production workloads. Synthetic tests at scale are instructive but nothing catches my attention like data from real, high-scale, production systems. Consequently, I really liked the disk population studies from Google and CMU at FAST2007 (Failure Trends in a Large Disk Population, Disk Failures in the Real World: What does a MTBF of 100,000 hours mean to you). These two papers presented actual results from independent production disk populations of 100,000 each. My quick summary of these 2 papers is basically “all you have learned about disks so far is probably wrong.”

Disk failures are often the #1 or #2 failing component in a storage system usually just ahead of memory. Occasionally fan failures lead disk but that isn’t the common case. We now have publically available data on disk failures but not much has been published on other component failure rates and even less on the overall storage stack failure rates. Cloud storage systems are multi-tiered, distributed systems involving 100s to even 10s of thousands of servers and huge quantities of software. Modeling the failure rates of discrete components in the stack is difficult but, with the large amount of component failure data available to large fleet operators, it can be done. What’s much more difficult to model are correlated failures.

Essentially, there are two challenges encountered when attempting to modeling overall storage system reliability: 1) availability of component failure data and 2) correlated failures. The former is available to very large fleet owners but is often unavailable publically. Two notable exceptions are disk reliability data from the two FAST’07 conference papers mentioned above. Other than these two data points, there is little credible component failure data publically available. Admittedly, component manufacturers do publish MTBF data but these data are often owned by the marketing rather than engineering teams and they range between optimistic and epic works of fiction.

Even with good quality component failure data, modeling storage system failure modes and data durability remains incredibly difficult. What makes this hard is the second issue above: correlated failure. Failures don’t always happen alone, many are correlated, and certain types of rare failures can take down the entire fleet or large parts of it. Just about every model assumes failure independence and then works out data durability to many decimal points. It makes for impressive models with long strings of nines but the challenge is the model is only as good as the input. And one of the most important model inputs is the assumption of component failure independence which is violated by every real-world system of any complexity. Basically, these failure models are good at telling you when your design is not good enough but they can never tell you how good your design actually is nor whether it is good enough.

Where the models break down is in modeling rare events and non-independent failures. The best way to understand common correlated failure modes is to study storage systems at scale over longer periods of time. This won’t help us understand the impact of very rare events. For example, Two thousand years of history would not helped us model or predict that a airplane would be flown into the World Trade Center. And certainly the odds of it happening again 16 min and 20 seconds later would be close to impossible. Studying historical storage system failure data will not help us understand the potential negative impact of very rare black swan events but it does help greatly in understanding the more common failure modes including correlated or non-independent failures.

Murray Stokely recently sent me Availability in Globally Distributed Storage Systems which is the work of a team from Google and Columbia University. They look at a high scale storage system at Google that includes multiple clusters of Bigtable which is layered over GFS which is implemented as a user–mode application over Linux file system. You might remember Stokely from my Using a post I did back in March titled Using a Market Economy. In this more recent paper, the authors study 10s of Google storage cells each of which is comprised of between 1,000 and 7,000 servers over a 1 year period. The storage cells studied are from multiple datacenters in different regions being used by different projects within Google.

I like the paper because it is full of data on a high-scale production system and it reinforces many key distributed storage system design lessons including:

· Replicating data across multiple datacenters greatly improves availability because it protects against correlated failures.

o Conclusion: Two way redundancy in two different datacenters is considerably more durable than 4 way redundancy in a single datacenter.

· Correlation among node failures dwarfs all other contributions to unavailability in production environments.

· Disk failures can result in permanent data loss but transitory node failures account for the majority of unavailability.

The abstract of the paper:

Highly available cloud storage is often implemented with complex, multi-tiered distributed systems built on top of clusters of commodity servers and disk drives. Sophisticated management, load balancing and recovery techniques are needed to achieve high performance and availability amidst an abundance of failure sources that include software, hardware, network connectivity, and power issues. While there is a relative wealth of failure studies of individual components of storage systems, such as disk drives, relatively little has been reported so far on the overall availability behavior of large cloud-based storage services. We characterize the availability properties of cloud storage systems based on an extensive one year study of Google's main storage infrastructure and present statistical models that enable further insight into the impact of multiple design choices, such as data placement and replication strategies. With these models we compare data availability under a variety of system parameters given the real patterns of failures observed in our fleet.

--jrh

James Hamilton

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

GPU Clusters in 10 Minutes

James Hamilton — Wed, 17 Nov 2010 14:15:32 GMT

Earlier this week Clay Magouyrk sent me a pointer to some very interesting work: A Couple More Nails in the Coffin of the Private Compute Cluster: Benchmarks for the Brand New Cluster GPU Instance on Amazon EC2.

This detailed article has detailed benchmark results from runs on the new Cluster GPU Instance type and leads in with:

During the past few years it has been no secret that EC2 has been best cloud provider for massive scale, but loosely connected scientific computing environments. Thankfully, many workflows we have encountered have performed well within the EC2 boundaries. Specifically, those that take advantage of pleasantly parallel, high-throughput computing workflows.> Still, the AWS approach to virtualization and available hardware has made it difficult to run workloads which required high bandwidth or low latency communication within a collection of distinct worker nodes. Many of the AWS machines used CPU technology that, while respectable, was not up to par with the current generation of chip architectures. The result? Certain use cases simply were not a good fit for EC2 and were easily beaten by in-house clusters in benchmarking that we conducted within the course of our research. All of that changed when Amazon released their Cluster Compute offering.>

The author goes on to run the Saleable Heterogeneous cOmputing BenChmarking Suite and compare EC2 with Native performance and conclude:

With this new AWS offering, the line between internal hardware and virtualized, cloud-based hardware for high performance computing using GPUs has indeed been blurred.

Finally a run with a Cycle Computing customer workload:

Based on the positive results of our SHOC benchmarking, we approached a Fortune 500 Life Science and a Finance/Insurance clients who develop and use their own GPU-accelerated software, to run their applications on the GPU-enabled Cluster Compute nodes. For both applications, the applications perform a large number of Monte Carlo simulations for given set of initial data, all pleasantly parallel. The results, similar to the SHOC result, were that the EC2 GPU-enabled Cluster Compute nodes performed as well as, or better than, the in-house hardware maintained by our clients.

Even if you only have a second, give the results a scan: http://blog.cyclecomputing.com/2010/11/a-couple-more-nails-in-the-coffin-of-the-private-compute-cluster-gpu-on-cloud.html.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

HPC in the Cloud with GPGPUs

James Hamilton — Mon, 15 Nov 2010 12:57:46 GMT

A year and half ago, I did a blog post titled heterogeneous computing using GPGPUs and FPGA. In that note I defined heterogeneous processing as the application of processors with different instruction set architectures (ISA) under direct application programmer control and pointed out that this really isn’t all that new a concept. We have had multiple ISAs in systems for years. IBM mainframes had I/O processors (Channel I/O Processors) with a different ISA than the general CPUs , many client systems have dedicated graphics coprocessors, and floating point units used to be independent from the CPU instruction set before that functionality was pulled up onto the chip. The concept isn’t new.

What is fairly new is 1) the practicality of implementing high-use software kernels in hardware and 2) the availability of commodity priced parts capable of vector processing. Looking first at moving custom software kernels into hardware, Field Programmable Gate Arrays (FPGA) are now targeted by some specialized high level programming languages. You can now write in a subset of C++ and directly implement commonly used software kernels in hardware. This is still the very early days of this technology but some financial institutions have been experimenting with moving computationally expensive financial calculations into FPGAs to save power and cost. See Platform-Based Electronic System-Level (ESL) Synthesis for more on this.

The second major category of heterogeneous computing is much further evolved and is now beginning to hit the mainstream. Graphics Processing Units (GPUs) essentially are vector processing units in disguise. They originally evolved as graphics accelerators but it turns out a general purpose graphics processor can form the basis of an incredible Single Instruction Multiple Data (SIMD) processing engine. Commodity GPUs have most of the capabilities of the vector units in early supercomputers. What’s missing is they have been somewhat difficult to program in that the pace of innovation is high and each model of GPU have differences in architecture and programming models. It’s almost impossible to write code that will directly run over a wide variety of different devices. And the large memories in these graphics processors typically are not ECC protected. An occasional pixel or two wrong doesn’t really matter in graphical output but you really do need ECC memory for server side computing.

Essentially we have commodity vector processing units that are hard to program and lack ECC. What to do? Add ECC memory and a abstraction layer that hides many of the device-to-device differences. With those two changes, we have amazingly powerful vector units available at commodity prices. One abstraction layer that is getting fairly broad pickup is Compute Unified Device Architecture or CUDA developed by NVIDIA. There are now CUDA runtime support libraries for many programming languages including C, FORTRAN, Python, Java, Ruby, and Perl.

Current generation GPUS are amazingly capable devices. I’ve covered the speeds and feeds of a couple in past postings: ATI RV770 and the NVIDIA GT200.

Bringing it all together, we have commodity vector units with ECC and an abstraction layer that makes it easier to program them and allows programs to run unchanged as devices are upgraded. Using GPUs to host general compute kernels is generically referred to as General Purpose Computation on Graphics Processing Units. So what is missing at this point? The pay-as-you go economics of cloud computing.

You may recall I was excited last July when Amazon Web Services announced the Cluster Compute Instance: High Performance Computing Hits the Cloud. The EC2 Cluster Compute Instance is capable but lacks a GPU:

· 23GB memory with ECC

· 64GB/s main memory bandwidth

· 2 x Intel Xeon X5570 (quad-core Nehalem)

· 2 x 845GB HDDs

· 10Gbps Ethernet Network Interface

What Amazon Web Services just announced is a new instance type with the same core instance specifications as the cluster compute instance above, the same high-performance network, but with the addition of two NVIDIA Tesla M2050 GPUs in each server. See supercomputing at 1/10^th the Cost. Each of these GPGPUs is capable of over 512 GFLOPs and so, with two of these units per server, there is a booming teraFLOP per node.

Each server in the cluster is equipped with a 10Gbps network interface card connected to a constant bisection bandwidth networking fabric. Any node can communicate with any other node at full line rate. It’s a thing of beauty and a forerunner of what every network should look like.

There are two full GPUs in each Cluster GPU instance each of which dissipates 225W TDP. This felt high to me when I first saw it but, looking at work done per watt, it’s actually incredibly good for workloads amenable to vector processing. The key to the power efficiency is the performance. At over 10x the performance of a quad core x86, the package is both power efficient and cost efficient.

The new cg1.4xlarge EC2 instance type:

· 2 x NVIDIA Tesla M2050 GPUs

· 22GB memory with ECC

· 2 x Intel Xeon X5570 (quad-core Nehalem)

· 2 x 845GB HDDs

· 10Gbps Ethernet Network Interface

With this most recent announcement, AWS now has dual quad core servers each with dual GPGPUs connected by a 10Gbps full-bisection bandwidth network for $2.10 per node hour. That’s $2.10 per teraFLOP. Wow.

· The Amazon Cluster GPU Instance type announcement: Announcing Cluster GPU Instances for EC2

· More information on the EC2 Cluster GPU instances: http://aws.amazon.com/ec2/hpc-applications/

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

AWS Compute Cluster #231 on Top 500

James Hamilton — Sun, 14 Nov 2010 19:21:19 GMT

The Top 500 Super Computer Sites list was just updated and AWS Compute Cluster is now officially in the top ½ of the list. That means when you line up all the fastest, multi-million dollar, government lab sponsored super computers from #1 through to #500, the AWS Compute Cluster Instance is at #231.

231

Amazon Web Services
United States

Amazon EC2 Cluster Compute Instances - Amazon EC2 Cluster, Xeon X5570 2.95 Ghz, 10G Ethernet / 2010
Self-made

7040

41.82

82.51

One of the fastest supercomputers in the world for $1.60/node hour. Cloud computing changes the economics in a pretty fundamental way.

More details at: High Performance Computing Hits the Cloud.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Datacenter Power Efficiency

James Hamilton — Fri, 12 Nov 2010 20:02:55 GMT

Kushagra Vaid presented at Datacenter Power Efficiency at Hotpower ’10. The URL to the slides is below and my notes follow. Interesting across the board but most notable for all the detail on the four major Microsoft datacenters:

· Services speeds and feeds:

o Windows LiveID: More than 1B authentications/day

o Exchange Hosted Services: 2 to 4B email messages per day

o MSN: 550M unique visitors monthly

o Hotmail: 400M active users

o Messenger: 320M active users

o Bing: >2B queries per month

· Microsoft Datacenters: 141MW total, 2.25M sq ft

o Quincy:

§ 27MW

§ 500k sq ft total

§ 54W/sq ft

o Chicago:

§ 60MW

§ 707k sq ft total

§ 85W/sq ft

§ Over $500M invested

§ $8.30/W (assuming $500M cost)

§ 3,400 tons of steel

§ 190 miles of conduit

§ 2,400 tons of copper

§ 26,000 yards of concrete

§ 7.5 miles of chilled water piping

§ 1.5M man hours of labor

§ Two floors of IT equipment:

· Lower floor: medium reliability container bay (standard IOS containers supplied by Dell)

· Upper floor: high reliability traditional colo facility

§ Note the picture of the upper floor shows colo cages suggesting that Microsoft may not be using this facility for their internal workloads.

§ PUE: 1.2 to 1.5

§ 3,000 construction related jobs

o Dublin:

§ 27MW

§ 570k sq ft total

§ 47W/sq ft

o San Antonio:

§ 27MW

§ 477k sq ft total

§ 57W/sq ft

· The power densities range between 45 and 85W/sq ft range which is incredibly low for relatively new builds. I would have expected something in the 200W/sq ft to 225W/sq ft range and perhaps higher. I suspect the floor space numbers are gross floor space numbers including mechanical, electrical, and office space rather than raised floor.

· Kushagra reports typical build costs range from $10 to $15/W

o based upon the Chicago data point, the Microsoft build costs are around $8/sq ft. Perhaps a bit more in the other facilities in that Chicago isn’t fully redundant on the lower floor whereas the generator count at the other facilities suggest they are.

· Gen 4 design

o Modular design but the use of ISO standard containers has been discontinued. However, they modules are prefabricated and delivered by flatbed truck

o No mechanical cooling

§ Very low water consumption

o 30% to 50% lower costs

o PUE of 1.05 to 1.1

· Analysis of AC vs DC power distribution concluding that DC more efficient at low loads and AC at high loads

o Over all the best designs are within 1 to 2% of each other

· Recommends higher datacenter temperatures

· Key observation:

o The best price/performance and power/performance is often found in lower-power processors

· Kushagra found substantial power savings using C6 and showed server idle power can be dropped to 31% of full load power

o Good servers typically run in the 45 to 50% range and poor designs can be as bad as 80%

The slides: http://mvdirona.com/jrh/TalksAndPapers/KushagraVaidMicrosoftDataCenters.pdf

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Datacenter Networks are in my Way

James Hamilton — Sun, 31 Oct 2010 19:30:28 GMT

I did a talk earlier this week on the sea change currently taking place in datacenter networks. In Datacenter Networks are in my Way I start with an overview of where the costs are in a high scale datacenter. With that backdrop, we note that networks are fairly low power consumers relative to the total facility consumption and not even close to the dominant cost. Are they actually a problem? The rest of the talk is arguing networks are actually a huge problem across the board including cost and power. Overall, networking gear lags behind the rest of the high-scale infrastructure world, block many key innovations, and actually are both cost and power problems when we look deeper.

The overall talk agenda:

· Datacenter Economics

· Is Net Gear Really the Problem?

· Workload Placement Restrictions

· Hierarchical & Over-Subscribed

· Net Gear: SUV of the Data Center

· Mainframe Business Model

· Manually Configured & Fragile at Scale

· New Architecture for Networking

In a classic network design, there is more bandwidth within a rack and more within an aggregation router than across the core. This is because the network is over-subscribed. Consequently, instances of a workload often needs to be placed topologically near to other instances of the workload, near storage, near app tiers, or on the same subnet. All these placement restrictions make the already over-constrained workload placement problem even more difficult. The result is either the constraints are not met which yields poor workload performance or the constraints are met but overall server utilization is lower due to accepting these constraints. What we want is all points in the datacenter equidistant and no constraints on workload placement.

Continuing on the over-subscription problem mentioned above, data intensive workloads like MapReduce and high performance computing workloads run poorly on oversubscribed networks. Its not at all uncommon for a MapReduce workload to transport the entire data set at least once over the network during job run. The cost of providing a flat, all-points-equidistant network are so high, that most just accept the constraint and other run MapReduce poorly or only run them in narrow parts of the network (accepting workload placement constraints).

Net gear doesn’t consume much power relative to total datacenter power consumption – other gear in the data center are, in aggregate much worse. However, network equipment power is absolutely massive today and it is trending up fast. A fully configured Cisco Nexus 7000 requires 8 circuits of 5kw each. Admittedly some of that power is for redundancy but how can 120 ports possibly require as much power provisioned as 4 average sized full racks of servers? Net gear is the SUV of the datacenter.

The network equipment business model is broken. We love the server business model where we have competition at the CPU level, more competition at the server level, and an open source solution for control software. In the networking world, it’s a vertically integrated stack and this slows innovation and artificially holds margins high. It’s a mainframe business model.

New solutions are now possible with competing merchant silicon from Broadcom, Marvell, and Fulcrum and competing switch designs built on all three. We don’t yet have the open source software stack but there are some interesting possibilities on the near term horizon with OpenFlow being perhaps the most interesting enabler. More on the business model and why I’m interested in OpenFlow at: Networking: The Last Bastion of the Mainframe Computing.

Talk slides: http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_POA20101026_External.pdf

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

AWS Free Tier: 750 hours of EC2 for free

James Hamilton — Fri, 22 Oct 2010 00:35:44 GMT

What happens when you really, really, focus on efficient infrastructure and driving down costs while delivering a highly available, high performance service? Well, it works. Costs really do fall and the savings can be passed on to customers.

AWS prices have been falling for years but this is different. Its now possible to offer an small EC2 instance for free. You can now have 750 hours of EC2 usage per month without charge.

The details:

· 750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) – enough hours to run continuously each month*

· 750 hours of an Elastic Load Balancer plus 15 GB data processing*

· 10 GB of Amazon Elastic Block Storage, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests*

· 5 GB of Amazon S3 storage, 20,000 Get Requests, and 2,000 Put Requests*

· 30 GB per of internet data transfer (15 GB of data transfer “in” and 15 GB of data transfer “out” across all services except Amazon CloudFront)*

· 25 Amazon SimpleDB Machine Hours and 1 GB of Storage**

· 100,000 Requests of Amazon Simple Queue Service**

· 100,000 Requests, 100,000 HTTP notifications and 1,000 email notifications for Amazon Simple Notification Service**

More information: http://aws.amazon.com/free/

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Amazon Web Services Book

James Hamilton — Tue, 19 Oct 2010 15:21:29 GMT

Long time Amazon Web Services Alum Jeff Barr has written a book on AWS. Jeff’s been with AWS since the very early days and he knows the services well. The new book Host Your Web Site in the Cloud: Amazon Web Services Made Easy, covers each of the major AWS services, how to write code against them, with code examples in PHP. It covers S3, EC2, SQS, EC2 Monitoring, Auto Scaling, Elastic Load Balancing, and SimpleDB. The table of contents:

Recommended if you are interested in Cloud Computing and AWS: http://www.amazon.com/Host-Your-Web-Site-Cloud/dp/0980576830.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Netflix Migration to the Cloud

James Hamilton — Sun, 10 Oct 2010 19:53:44 GMT

This morning I came across an article written by Sid Anand, an architect at Netflix that is super interesting. I liked it for two reasons: 1) it talks about the move of substantial portions of a high-scale web site to the cloud, some of how it was done, and why it was done, and 2) its gives best practices on AWS SimpleDB usage.

I love articles about how high scale systems work. Some past postings:

The article starts off by explaining why Netflix decided to move their infrastructure to the cloud:

Circa late 2008, Netflix had a single data center. This single data center raised a few concerns. As a single-point-of-failure, it represented a liability – data center outages meant interruptions to service and negative customer impact. Additionally, with growth in both streaming adoption and subscription levels, Netflix would soon outgrow this data center – we foresaw an imminent need for more power, better cooling, more space, and more hardware.

Our option was to build more data centers. Aside from high upfront costs, this endeavor would likely tie up key engineering resources in data center scale out activities, making them unavailable for new product initiatives. Additionally, we recognized the management of multiple data centers to be a complex task. Building out and managing multiple data centers seemed a risky distraction.

Rather than embarking on this path, we chose a more radical one. We decided to leverage one of the leading IAAS offerings at the time, Amazon Web Services. With multiple, large data centers already in operation and multiple levels of redundancy in various web services (e.g. S3 and SimpleDB), AWS promised better availability and scalability in a relatively short amount of time.

By migrating various network and back-end operations to a 3^rd party cloud provider, Netflix chose to focus on its core competency: to deliver movies and TV shows.

I’ve read considerable speculation over the years on the difficulty of moving to cloud services. Some I agree with – these migrations do take engineering investment – while other reports seem to less well thought through focusing mostly on repeating concerns speculated upon by others. Often, the information content is light.

I know the move to the cloud can be done and is done frequently because, where I work, I’m lucky enough to see it happen every day. But the Netflix example is particularly interesting in that 1) Netflix is a fairly big enterprise with a market capitalization of $7.83B – moving this infrastructure is substantial and represents considerable complexity. It is a great example of what can be done; 2) Netflix is profitable and has no economic need to make the change – they made the decision to avoid distraction and stay focused on the innovation that made the company as successful as it is; and 3) they are willing to contribute their experiences back to the community. Thanks to Sid and Netflix for the later.

For more detail, check out the more detailed document that Sid Anand has posted: Netflix’s Transition to High-Availability Storage Systems.

For those SimpleDB readers, here’s a set of SimpleDB best practices from Sid’s write-up:

· Partial or no SQL support. Loosely-speaking, SimpleDB supports a subset of SQL

o Do GROUP BY and JOIN operations in the application layer

o One way to avoid the need for JOINs is to denormalize multiple Oracle tables into a single logical SimpleDB domain.

· No relations between domains

o Compose relations in the application layer

· No transactions

o Use SimpleDB’s Optimistic Concurrency Control API: ConditionalPut and ConditionalDelete

· No Triggers

o Do without

· No PL/SQL

o Do without

· No schema - This is non-obvious. A query for a misspelled attribute name will not fail with an error

o Implement a schema validator in a common data access layer

· No sequences

o Sequences are often used as primary keys

§ In this case, use a naturally occurring unique key. For example, in a Customer Contacts domain, use the customer’s mobile phone number as the item key

§ If no naturally occurring unique key exists, use a UUID

o Sequences are also often used for ordering

§ Use a distributed sequence generator

· No clock operations

o Do without

· No constraints. Specifically, no uniqueness, no foreign key, no referential & no integrity constraints

o Application can check constraints on read and repair data as a side effect. This is known as read-repair. Read repair should use the CondtionalPut or ConditionalDelete API to make atomic changes

If you are interested high sale web sites or cloud computing in general, this one is worth a read: Netflix’s Transition to High-Availability Storage Systems.

--jrh

Update: Netflix architects Adrian Cockcroft and Sid Anand are both presenting at QconSF between November 1^st and 5^th in San Francisco: http://qconsf.com/sf2010. > >
James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Scaling AWS Relational Database Service

James Hamilton — Sat, 09 Oct 2010 18:05:02 GMT

Hosting multiple MySQL engines with MySQL Replication between them is a common design pattern for scaling read-heavy MySQL workloads. As with all scaling techniques, there are workloads for which it works very well but there are also potential issues that need to be understood. In this case, all write traffic is directed to the primary server and, consequently is not scaled which is why this technique works best for workloads heavily skewed towards reads. But, for those fairly common read heavy workloads, the techniques works very well and allows scaling the read workload across over a fleet of MySQL instances. Of course, as with any asynchronous replication scheme, the read replicas are not transactionally updated. So any application running on MySQL read replica’s must be tolerant of eventually consistent updates.

Load balancing high read traffic over multiple MySQL instances works very well but this is only one of the possible tools used to scale this type of workload. Another very common technique is to put a scalable caching layer in front of the relational database fleet. By far the most common caching layer used by high-scale services is Memcached.

Another database scaling technique is to simply not use a relational database. For workloads that don’t need schema enforcement and complex query, NoSQL databases offer both a cheaper and a simpler approach to hosting the workload. SimpleDB is the AWS hosted NoSQL database with Netflix being one of the best known users (slide deck from Netflix’s Adrian Cockcroft: http://www.slideshare.net/adrianco/netflix-oncloudteaser). Cassandra is another common RDBMS alternative in heavy use by many high-scale sites including Facebook where it was originally conceived. Cassandra is also frequently run on AWS with the Cassandra Wiki offering scripts to make it easy install and configure on Amazon EC2.

For those workloads where a relational database is the chosen solution, MySQL read replication is a good technique to have in your scaling tool kit. Last week Amazon announced read replica support for the AWS Relational Database Service. The press release is at: Announcing Read Replicas, Lower High Memory DB Instance Price for Amazon AWS.

You can now create one or more replicas of a given “source” DB Instance and serve incoming read traffic from multiple copies of your data. This new database deployment option enables you to elastically scale out beyond the capacity constraints of a single DB Instance for read-heavy database workloads. You can use Read Replicas in conjunction with Multi-AZ replication for scalable, reliable, and highly available production database deployments.

If you are running MySQL and wish you had someone else to manage it for you, check out Amazon RDS. The combination of read replicas to scale read workloads and Multi-AZ support for multi-data center, high availability make it a pretty interesting way to run MySQL.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Overall Data Center Costs

James Hamilton — Sat, 18 Sep 2010 20:56:19 GMT

A couple of years ago, I did a detailed look at where the costs are in a modern , high-scale data center. The primary motivation behind bringing all the costs together was to understand where the problems are and find those easiest to address. Predictably, when I first brought these numbers together, a few data points just leapt off the page: 1) at scale, servers dominate overall costs, and 2) mechanical system cost and power consumption seems unreasonably high. Both of these areas have proven to be important technology areas to focus upon and there has been considerable industry-wide innovation particularly in cooling efficiency over the last couple of years.

I posted the original model at the Cost of Power in Large-Scale Data Centers. One of the reasons I posted it was to debunk the often repeated phrase “power is the dominate cost in a large-scale data center”. Servers dominate with mechanical systems and power distribution close behind. It turns out that power is incredibly important but it’s not the utility kWh charge that makes power important. It’s the cost of the power distribution equipment required to consume power and the cost of the mechanical systems that take the heat away once the power is consumed. I referred to this as fully burdened power.

Measured this way, power is the second most important cost. Power efficiency is highly leveraged when looking at overall data center costs, it plays an important role in environmental stewardship, and it is one of the areas where substantial gains continue to look quite attainable. As a consequence, this is where I spend a considerable amount of my time – perhaps the majority – but we have to remember that servers still dominate the overall capital cost.

This last point is a frequent source of confusion. When server and other IT equipment capital costs are directly compared with data center capital costs, the data center portion actually is larger. I’ve frequently heard “how can the facility cost more than the servers in the facility – it just doesn’t make sense.” I don’t know whether or not it makes sense but it actually is not true at this point. I could imagine the infrastructure costs one day eclipsing those of servers as server costs continue to decrease but we’re not there yet. The key point to keep in mind is the amortization periods are completely different. Data center amortization periods run from 10 to 15 years while server amortizations are typically in the three year range. Servers are purchased 3 to 5 times during the life of a datacenter so, when amortized properly, they continue to dominate the cost equation.

In the model below, I normalize all costs to a monthly bill by taking consumable like power and billing them monthly by consumption and taking capital expenses like servers, networking or datacenter infrastructure, and amortizing over their useful lifetime using a 5% cost of money and, again, billing monthly. This approach allows us to compare non-comparable costs such as data center infrastructure with servers and networking gear each with different lifetimes. The model includes all costs “below the operating system” but doesn’t include software licensing costs mostly because open source is dominant in high scale centers and partly because licensing costs very can vary so widely. Administrative costs are not included for the same reason. At scale, hardware administration, security, and other infrastructure-related people costs disappear into the single digits with the very best services down in the 3% range. Because administrative costs vary so greatly, I don’t include them here. On projects with which I’ve been involved, they are insignificantly small so don’t influence my thinking much. I’ve attached the spreadsheet in source form below so you can add in factors such as these if they are more relevant in your environment.

Late last year I updated the model for two reasons: 1) there has been considerable infrastructure innovation over the last couple of years and costs have changed dramatically during that period and 2) because of the importance of networking gear to the cost model, I factor out networking from overall IT costs. We now have IT costs with servers and storage modeled separately from networking. This helps us understand the impact of networking on overall capital cost and on IT power.

When I redo these data, I keep the facility server count in the 45,000 to 50,000 server range. This makes it an reasonable scale facility –big enough to enjoy the benefits of scale but nowhere close to the biggest data centers. Two years ago, 50,000 servers required a 15MW facility (25MW total load). Today, due to increased infrastructure efficiency and reduced individual server power draw, we can support 46k servers in an 8MW facility (12MW total load). The current rate of innovation in our industry is substantially higher than it has been any time in the past with much of this innovation driven by mega service operators.

Keep in mind, I’m only modeling those techniques well understood and reasonably broadly accepted as good quality data center design practices. Most of the big operators will be operating at efficiency levels far beyond those used here. For example, in this model we’re using a Power Usage Efficiency (PUE) of 1.45 but Google, for example, reports PUE across the fleet of under 1.2: Data Center Efficiency Measurements. Again, the spread sheet source is attached below so feel free to change to the PUE used by the model as appropriate.

These are the assumptions used by this year’s model:

Using these assumptions we get the following cost structure:

For those of you interested in playing with different assumptions, the spreadsheet source is here: http://mvdirona.com/jrh/TalksAndPapers/PerspectivesDataCenterCostAndPower.xls.

If you choose to use this spreadsheet directly or the data above, please reference the source and include the URL to this pointing.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Amazon EC2 for under $15/Month

James Hamilton — Thu, 09 Sep 2010 12:53:17 GMT

You can now run an EC2 instance 24x7 for under $15/month with the announcement last night of the Micro instance type. The Micro includes 613 MB of RAM and can burst for short periods up to 2 EC2 Compute Units(One EC2 Compute Unit equals 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor). They are available in all EC2 supported regions and in 32 or 64 bit flavors.

The design point for Micro instances is to offer a small amount of consistent, always available CPU resources while supporting short bursts above this level. The instance type is well suited for lower throughput applications and web sites that consume significant compute cycles but only periodically.

Micro instances are available for Windows or Linux with Linux at $0.02/hour and windows at $0.03/hour. You can quickly start a Micro instance using the EC2 management console: EC2 Console.

The announcement: Announcing Micro Instances for Amazon EC2

More information: Amazon Elastic Compute Cloud (Amazon EC2)

--jrh