Finally! I’ve been dying to talk about DynamoDB since work began on this scalable, low-latency, high-performance NoSQL service at AWS. This morning, AWS announced availability of DynamoDB: Amazon Web Services Launches Amazon DynamoDB – A New NoSQL Database Service Designed for the Scale of the Internet.

In a past blog entry, One Size Does Not Fit All, I offered a taxonomy of 4 different types of structured storage system, argued that Relational Database Management Systems are not sufficient, and walked through some of the reasons why NoSQL databases have emerged and continue to grow market share quickly. The four database categories I introduced were: 1) features-first, 2) scale-first, 3) simple structure storage, and 4) purpose-optimized stores. RDBMS own the first category.

DynamoDB targets workloads fitting into the Scale-First and Simple Structured storage categories where NoSQL database systems have been so popular over the last few years.  Looking at these two categories in more detail, Scale-First is:

Scale-first applications are those that absolutely must scale without bound and being able to do this without restriction is much more important than more features. These applications are exemplified by very high scale web sites such as Facebook, MySpace, Gmail, Yahoo, and Amazon.com. Some of these sites actually do make use of relational databases but many do not. The common theme across all of these services is that scale is more important than features and none of them could possibly run on a single RDBMS. As soon as a single RDBMS instance won’t handle the workload, there are two broad possibilities: 1) shard the application data over a large number of RDBMS systems, or 2) use a highly scalable key-value store.

And, Simple Structured Storage:

There are many applications that have a structured storage requirement but they really don’t need the features, cost, or complexity of an RDBMS. Nor are they focused on the scale required by the scale-first structured storage segment. They just need a simple key value store. A file system or BLOB-store is not sufficiently rich in that simple query and index access is needed but nothing even close to the full set of RDBMS features is needed. Simple, cheap, fast, and low operational burden are the most important requirements of this segment of the market.

More detail at: One Size Does Not Fit All.

The DynamoDB service is a unified purpose-built hardware platform and software offering. The hardware is based upon a custom server design using Flash Storage spread over a scalable high speed network joining multiple data centers.

DynamoDB supports a provisioned throughput model. A DynamoDB application programmer decides the number of database requests per second their application should be capable of supporting and DynamoDB automatically spreads the table over an appropriate number of servers. At the same time, it also reserves the required network, server, and flash memory capacity to ensure that request rate can be reliably delivered day and  night, week after week, and year after year.  There is no need to worry about a neighboring application getting busy or running wild and taking all the needed resources. They are reserved and there whenever needed.

The sharding techniques needed to achieve high requests rates are well understood industry-wide but implementing them does take some work. Reliably reserving capacity so it is always there when you need it, takes yet more work.  Supporting the ability to allocate more resources, or even less, while online and without disturbing the current request rate takes still more work. DynamoDB makes all this easy. It supports online scaling between very low transaction rates to applications requiring millions of requests per second. No downtime and no disturbance to the currently configured application request rate while resharding. These changes are done online only by changing the DynamoDB provisioned request rate up and down through an API call.

In addition to supporting transparent, on-line scaling of provisioned request rates up and down over 6+ orders of magnitude with resource reservation, DynamoDB is also both consistent and multi-datacenter redundant. Eventual consistency is a fine programming model for some applications but it can yield confusing results under some circumstances. For example, if you set a value to 3 and then later set it to 4, then read it back, 3 can be returned. Worse, the value could be set to 4, verified to be 4 by reading it, and yet 3 could be returned later. It’s a tough programming model for some applications and it tends to be overused in an effort to achieve low-latency and high throughput.  DynamoDB avoids forcing this by supporting low-latency and high throughout while offering full consistency. It also offers eventual consistency at lower request cost for those applications that run well with that model. Both consistency models are supported.

It is not unusual for a NoSQL store to be able to support high transaction rates. What is somewhat unusual is to be able to scale the provisioned rate up and down while on-line. Achieving that while, at the same time, maintaining synchronous, multi-datacenter redundancy is where I start to get excited.

Clearly nobody wants to run the risk of losing data but NoSQL systems are scale-first by definition. If the only way to high throughput and scale, is to run risk and not commit the data to persistent storage at commit time, that is exactly what is often done. This is where  DynamoDB really shines. When data is sent to DynamoDB, it is committed to persistent and reliable storage before the request is acknowledged. Again this is easy to do but doing it with average low single digit millisecond latencies is both harder and requires better hardware. Hard disk drives can’t do it and in-memory systems are not persistent so flash memory is the most cost effective solution.

But what if the server to which the data was committed fails, or the storage fails, or the datacenter is destroyed? On most NoSQL systems you would lose your most recent changes.  On the better implementations, the data might be saved but could be offline and unavailable. With dynamoDB, if data is committed just as the entire datacenter burns to the ground, the data is safe, and the application can continue to run without negative impact at exactly the same provisioned throughput rate. The loss of an entire datacenter isn’t even inconvenient (unless you work at Amazon :-)) and has no impact on your running application performance.

Combining rock solid synchronous, multi-datacenter redundancy with average latency in the single digits, and throughput scaling to the millions of requests per second is both an excellent engineering challenge and one often not achieved.

Relational systems remain an excellent solution for applications requiring Feature-First structured storage. AWS Relational Database Service supports both the MySQL and Oracle and relational database management systems: http://aws.amazon.com/rds/.

Just as I was blown away when I saw it possible to create the world’s 42^nd most powerful super computer with a few API calls to AWS (42: the Answer to the Ultimate Question of Life, the Universe and Everything), it is truly cool to see a couple of API calls to DynamoDB be all that it takes to get a scalable, consistent, low-latency, multi-datacenter redundant, NoSQL service configured, operational and online.

                                                --jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

Netflix is super interesting in that they are running at extraordinary scale, are a leader in the move to the cloud, and Adrian Cockcroft, the Netflix Director of Cloud Architecture, is always interesting in presentations. In this presentation Adrian covers similar material to his HPTS 2011 talk I saw last month.

His slides are up at: http://www.slideshare.net/adrianco/global-netflix-platform and my rough notes follow:

·         Netflix has 20 milion streaming members

o    Currently in US, Canada, and Latin America

o    Soon to be in UK and Ireleand

·         Netflix is 100% public cloud hosted

·         Why did Netflix move from their own high-scale facility to a public cloud?

o    Better business agility

o    Netflix was unable build datacenters fast enough

o    Capacity growth was both accelerating and unpredictable

o    Product launch spikes require massive new capacity (iPhone, Wii, PS3, & Xbox)

Netflix grew 37x from Jan 2010 through Jan 2011

·         Why did Netflix choose AWS as their cloud solution?

o    Chose AWS using Netflix own platform and tools

o    Netflix has unique platform requirements and extreme scale needing both agility & flexibility

o    Chose AWS partly because it was the biggest public cloud

§  Wanted to leverage AWS investment in features and automation

§  Wanted to use AWS availability zones and regions for availability, scalability, and global deployment

§  Didn’t want to be the biggest customer on a small cloud

o    But isn’t Amazon a competitor?

§  Many products that compete with Amazon run on AWS

§  Netflix is the “poster child” for the AWS Architecture

§  One of the biggest AWS customers

§  Netflix strategy: turn competitors into partners

o    Could Netflix use a different cloud from AWS

§  Would be nice and Netflix already uses 3 interchangeable CDN vendors

§  But no one else has the scale and features of AWS

·         “you have to be tall to ride this ride”

·         Perhaps in 2 to 3 years?

o    “We want to use cloud, we don’t want to build them”

§  Public clouds for agility and scale

§  We use electricity too but we don’t want to build a power station

§  AWS because they are big enough to allocated thousands of instances per hour when needed

o    Netflix Global PaaS

§  Supports all AWS Availability Zones and Regions

§  Supports multiple AWS accounts (test, prod, etc.)

§  Supports cross Regions and cross account data replication & archiving

§  Supports fine grained security with dynamic AWS keys

§  Autoscales to thousands of instances

§  Monitoring for millions of metrics

o    Portals and explorers:

§  Netflix Application Console (NAC): Primary AWS provisioning & config interface

§  AWS Usage Analyzer: cost breakdown by application and resource

§  SimpleDB Explorer: browse domains, items, attributes, values,…

§  Cassandra Explorer: browse clusters, keyspaces, column families, …

§  Base Service Explorer: browse endpoints, configs, perf metrics, …

o    Netflix Platform Services:

§  Discovery: Service Register for applications

§  Introspections: Endpoints

§  Cryptex: Dynamic security key management

§  Geo: Geographic IP lookup engine

§  Platform Serivce: Dynamic property configuration

§  Localization: manage and lookup local translations

§  EVcache: Eccetric Volatile (mem)Cached

§  Cassadra: Persistence

§  Zookeeper: Coordination

o    Netflix Persistence Services:

§  SimpleDB: Netflix moving to Cassandra

·         Latencies typically over 10msec

§  S3: using the JetS3t based interface with Netflix changes and updates

§  Eccentric Volatile Cache (evcache)

·         Discovery aware memcached based backend

·         Client abstractions for zone aware replication

·         Supports option to write to all zones, fast read from local

·         On average, latencies of under 1 msec

§  Cassandra

·         Chose because they value availability over consistency

·         On average, latency of “few microseconds”

§  MongoDB

§  MySQti: supports hard to scale, legacy, and small relational models

o    Implemented a Multi-Regional Data Replication system:

§  Oracle to SimpleDB and queued reverse path usingj SQS

o    High Availability:

§  Cassandra stores 3 local copies, 1 per availability zone

§  Each AWS availability zone is a separate building with separate power etc. but still fairly close together so synchronous access is practical

§  Synchronous access, durable, and highly available

Adrian’s slide deck is posted at: http://www.slideshare.net/adrianco/global-netflix-platform.

                                --jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

I seldom write consumer product reviews and this blog is about the furthest thing from a consumer focused site but, every so often, I come across a notable tidbit that is worthy of mention. A few weeks ago, it was Sprint unilaterally changing the terms of their wireless contracts (Sprint is Giving Free Customer Service Lessons). It just seemed a sufficiently confused decision that it was worthy of mention.

Here’s one that just nails it on the other side of the equation by obsessing over the customer experience: Roku. I’ve long known about Roku but I’m not a huge TV watcher so I’ve only been peripherally interested in the product. But we are both Netflix and Amazon Prime Instant Video customers and Roku supports both. And the entry level Roku streaming appliance is only $49 so we figured let’s give it a try. It actually ended up a bit more than $49 in that we first managed to upsell ourselves to a $59 Roku 2 to get HD, and then to a $79 device to get 1080P and then to a $99 device to het 1080P HD with a hardwired Ethernet connection. So we ended up with a $100 device. I think $50 is close to where this class of devices needs to end up but $100 is reasonable as well.

The device is amazing and shows what can be done with a focus on clean industrial design. It is incredibly small at only 3” square. I plugged it in, it booted up, updated its software, found its remote, upgraded the software on the remote and went live without any user interaction. I setup a Roku account, linked my Amazon account for access to Prime Instant Video, linked our Netflix account and it was ready to go.

The device is tiny, produces close to no heat, you don’t have to read the manual, the user interface is clean and notable for its snappiness. I expected a sluggish UI as many companies scrimp on processing power to get costs down but it is very snappy. In fact Netflix on a Roku is faster than the same support on an Xbox. The UI is clean, simple, snappy, and very elegant.

I love where consumer appliances are heading: simple, cheap, dedicated, purpose-build devices with clean user interfaces, and the hybrid delivery model where the user interface is delivered by the appliance but most of the functionality is hosted in the cloud.  The combination of cheap microelectronics, open source operating systems, and cloud hosting allows incredibly high function devices to be delivered at low cost. 

The Kindle Fire takes the hybrid cloud connected model a long way where the Fire’s Silk browser UI runs directly on the device close to the user where it can be highly interactive and responsive. But the power and network-bandwidth hungry browser backend is hosted on Amazon EC2 where connectivity is awesome and compute power is not battery constrained. I love the hybrid model and we are going to see more and more devices delivering a hybrid user experience where the compute intensive components are cloud hosted and user interface is in the device. My belief is that this is the future of consumer electronics and, as prices drop to the $30 to $50 range, everyone will have 10s of these special-purpose, cloud-connected devices.

For the first time in my life, I’m super interested in consumer devices and the possibilities of what can be done in the hybrid cloud-connected appliance model.

                                --jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

Yesterday the Top 500 Supercomputer Sites was announced. The Top500 list shows the most powerful commercially available supercomputer systems in the world. This list represents the very outside of what supercomputer performance is possible when cost is no object. The top placement on the list is always owned by a sovereign funded laboratory. These are the systems that only government funded agencies can purchase. But they have great interest for me because, as the cost of computing continues to fall, these performance levels will become commercially available to companies wanting to run high scale models and data intensive computing. In effect, the Top500 predicts the future so I’m always interested in the systems on the list.

What makes this list of the fastest supercomputers in the world released yesterday particularly unusual can be found at position #42.  42 is an anomaly of the first order. In fact, #42 is an anomaly across enough dimensions that its worth digging much deeper.

Virtualization Tax is Now Affordable:

I remember reading through the detailed specifications when the Cray 1 supercomputer was announced and marveling that it didn’t even use virtual memory. It was believed at the time that only real-mode memory access could deliver the performance needed.

We have come a long way in the nearly 40 years since the Cray 1 was announced. This #42 result was run not just using virtual memory but with virtual memory in a guest operating system running under a hypervisor. This is the only fully virtualized, multi-tenant super computer on the Top500 and it shows what is possible as the virtualization tax continues to fall. This is an awesome result and many more virtualization improvements are coming over the next 2 to 3 years.

Commodity Networks can Compete at the Top of the Performance Spectrum:

This is the only Top500 entrant below number 128 on the list that is not running either Infiniband or a proprietary, purpose-built network. This result at #42 is an all Ethernet network showing that a commodity network, if done right, can produce industry leading performance numbers.

What’s the secret?  10Gbps directly the host is the first part. The second is full non-blocking networking fabric where all systems can communicate at full line rate at the same time.  Worded differently, the network is not oversubscribed. See Datacenter Networks are in my Way for more on the problems with existing datacenter networks.

Commodity Ethernet networks continue to borrow more and more implementation approaches and good network architecture ideas from Infiniband, scale economics continues to drive down costs so non-blocking networks are now practical and affordable, and scale economics are pushing rapid innovation. Commodity equipment in a well-engineered overall service is where I see the future of networking continuing to head.

Anyone can own a Supercomputer for an hour:

You can’t rent supercomputing time by the hour from Lawrence Livermore National Laboratory. Sandia is not doing it either. But you can have a top50 supercomputer for under $2,600/hour. That is one of the world’s most powerful high performance computing systems  with 1,064 nodes and 8,512 cores for under $3k/hour. For those of you not needing quite this much power at one time, that’s $0.05/core hour which is ½ of the previous Amazon Web Services HPC system cost.

Single node speeds and feeds:

·         Processors: 8-core, 2 socket Intel Xeon @ 2.6 Ghz with hyperthreading

·         Memory: 60.5GB

·         Storage: 3.37TB direct attached and Elastic Block Store for remote storage

·         Networking: 10Gbps Ethernet with full bisection bandwidth within the placement group

·         Virtualized: Hardware Assisted Virtualization

·         API: cc2.8xlarge

Overall Top500 Result:

·         1064 nodes of cc2.8xlarge

·         240.09 TFlops at an excellent 67.8% efficiency

·         $2.40/node hour on demand

·         10Gbps non-blocking Ethernet networking fabric

Database Intensive Computing:

This is a database machine masquerading as a supercomputer. You don’t have to use the floating point units to get full value from renting time on this cluster. It’s absolutely a screamer as an HPC system. But it also has the potential to be the world’s highest performing MapReduce system (Elastic Map Reduce) with a full bisection bandwidth 10Gbps network directly to each node.  Any database or general data intensive workload with high per-node computational costs and/or high inter-node traffic will run well on this new instance type. 

If you are network bound, compute bound, or both, the EC2 cc2.8xlarge instance type could be the right answer. And, the amazing thing is that the cc2 instance type is ½ the cost per core of the cc1 instance.

Supercomputing is now available to anyone for $0.05/core hour. Go to http://aws.amazon.com/hpc-applications/ and give it a try. You no longer need to be a national lab or a government agency to be able run one of the biggest supercomputers in the world.

                                                                --jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

One of the talks that I particularly enjoyed yesterday at HPTS 2011 was Storage Infrastructure Behind Facebook Messages by Kannan Muthukkaruppan. In this talk, Kannan talked about the Facebook store for chats, email, SMS, & messages.

This high scale storage system is based upon HBase and Haystack. HBase is a non-relational, distributed database very similar to Google’s Big Table. Haystack is simple file system designed by Facebook for efficient photo storage and delivery. More on Haystack at: Facebook Needle in a Haystack.

In this Facebook Message store, Haystack is used to store attachments and large messages.  HBase is used for message metadata, search indexes, and small messages (avoiding the second I/O to Haystack for small messages like most SMS).

Facebook Messages takes 6B+ messages a day. Summarizing HBase traffic:

·         75B+ R+W ops/day with 1.5M ops/sec at peak

·         The average write operation inserts 16 records across multiple column families

·         2PB+ of cooked online data in HBase. Over 6PB including replication but not backups

·         All data is LZO compressed

·         Growing at 250TB/month

The Facebook Messages project timeline:

·         2009/12: Project started

·         2010/11: Initial rollout began

·         2011/07: Rollout completed with 1B+ accounts migrated to new store

·         Production changes:

o   2 schema changes

o   Upgraded to Hfile 2.0

They implemented a very nice approach to testing where, prior to release, they shadowed the production workload to the test servers.

After going into production the continued the practice of shadowing the real production workload into the test cluster to test before going into production:

The list of scares and scars from Kannan:

·         Not without our share of scares and incidents:

o   s/w bugs. (e.g., deadlocks, incompatible LZO used for bulk imported data, etc.)

§  found a edge case bug in log recovery as recently as last week!

·         performance spikes every 6 hours (even off-peak)!

o   cleanup of HDFS’s Recycle bin was sub-optimal! Needed code and config fix.

·         transient rack switch failures

·         Zookeeper leader election took than 10 minutes when one member of the quorum died. Fixed in more recent version of ZK.

·         HDFS Namenode – SPOF

·         flapping servers (repeated failures)

·         Sometimes, tried things which hadn’t been tested in dark launch!

o   Added a rack of servers to help with performance issue

§  Pegged top of the rack network bandwidth!

§  Had to add the servers at much slower pace. Very manual .

§  Intelligent load balancing needed to make this more automated.

·         A high % of issues caught in shadow/stress testing

·         Lots of alerting mechanisms in place to detect failures cases

o   Automate recovery for a lots of common ones

o   Treat alerts on shadow cluster as hi-pri too!

·         Sharding service across multiple HBase cells also paid off

Kannan’s slides are posted at: http://mvdirona.com/jrh/TalksAndPapers/KannanMuthukkaruppan_StorageInfraBehindMessages.pdf

                                                                --jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

Rough notes from a talk on COSMOS, Microsoft’s internal Map reduce systems from HPTS 2011. This is the service Microsoft uses internally to run MapReduce jobs. Interesting, Microsoft plans to use Hadoop in the external Azure service even though COSMOS looks quite good: Microsoft Announces Open Source Based Cloud Service. Rough notes below:

Talk: COSMOS: Big Data and Big Challenges

Speaker: Ed Harris

·         Petabyte storage and computation systems

·         Used primarily by search and advertising inside Microsoft

·         Operated as a service with just over 4 9s of availability

·         Massively parallel processing based upon Dryad

o   Dryad is very similar to MapReduce

·         Use SCOPE (structured Computation Optimized for Parallel Execution) over Dryad

o   A SQL-like language with an optimizers implemented over Dryad

·         They run hundreds of virtual clusters. In this model, internal Microsoft teams buy servers and given them to COSMOS and are subsequently assured at least these resources

o   Average 85% CPU over the cluster

·         Ingest 1 to 2 PB/day

·         Roughly 30% of the Search fleet is running COSMOS

·         Architecture:

o   Store Layer

§  Many extent nodes store and compress streams

§  Streams are sequences of extents

§  CSM: Cosmos Store Layer handles names, streams, and replication

·         First level compression is light. Data that is kept more than a week is more aggressively compressed after a week on the assumption that data that lives a week will likely live longer

o   Execution Layer:

§  Jobs queue up on virtual clusters and then executed

o   SCOPE Layer

§  Compiler and optimizer for SCOPE

§  Ed said that the optimizer is a branch of the SQL Server optimizer

·         They have 60+ Phd internships each year and hire ~30 a year

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

Last night EMC Chief Executive Joe Tucci laid out his view of where the information processing world is going over the next decade and where EMC will focus.  His primary point was cloud computing is the future and big data is the killer app for the cloud. He laid out the history of big transitions in our industry and argued the big discontinuities were always driven by a killer application. He sees the cloud as the next big and important transition for our industry.

This talk was presented as part of the University of Washington Distinguished Lecturer Series. With six TV cameras covering the action, there were nearly as many as some University of Washington Huskies games and it was well attended. The next talk in the series will be Bill Gates on October 27 presenting The Opportunity Ahead: A Conversation with Bill Gates. I’ll be presenting Internet Scale Storage on November 1^st.

If you are interested in any of the talks in the series, all are open to the public and the upcoming schedule is posted at: http://www.cs.washington.edu/news/newdlshome.html.

The most notable statistic from the Joe Tucci talk was the massive investment that EMC is making mergers and acquisitions. He said over the next 5 years, EMC will spend $10.5B in R&D – this number alone is amazingly large -- but what I found really startling was they expect to spend even more purchasing companies. They expect to spend $14.0B on M&A during this same period. That’s nearly $3B/year from just a single company. Amazing.

With many large companies increasingly looking to the startup community for new ideas and innovation, there is incredible opportunity for startups.  Joe emphasized the opportunity, saying that Washington in general and especially the University of Washington will likely be the source of many of these new companies. As large companies lean more on the startup community for new ideas, products, and services, it’s a good time to be starting a company.

My rough notes from the talk:

·         IDC reports:

o   This decade WW information content will grow 44x (0.9 zettabytes to 35.2)

o   90% unstructured

·         Big data has arrived

o   Mobile sensors

o   Social media

o   Video surveillance

o   Smart grids

o   Gene sequencing

o   Medical imaging

o   Geophysical exploration

·         73% maintaining existing infrastructure (true for 10 years)

o   JRH: I’ve heard this statistic before but it seems like nearly has to be the case the most companies are spending at least 3/4s of their investment continuing to running the business and around a ¼ on new applications. The statistic is usually presented as a problem but it feels like it might be close to the right ratio.

·         3D movie is about a petabyte with all camera angles and footage included

·         The average company is attacked 300 times per week

o   All CIO say this is way light – my home router gets nailed that many times in a good hour

·         IT staffing will increase less than 50% in next 10 years but the data under management will grow faster.

o   JRH: Again this seems like the desirable outcome where the data under management should be able to grow far faster than administrative team

·         EMCs Mission: To lead customers towards a hybrid cloud

o   Leading customers to x86 based private clouds and hybrid clouds

o   Burst, test & development, etc. into the public cloud

o   Hybrid cloud between private and public is the “big winner”

·         VM is basically a cloud operating systems

o   EMC still owns 80% of VMWare

o   There are now more than virtual machines shipped than physical machines

o   62% virtualized out of the gate

·         Applications like SAP, Oracle, and Microsoft are now available in the cloud

·         Killer app for the cloud is big data

o   Real time data analytics

·         New end user computing

o   IOS devices, android, windows, …

·         Tenets of cloud computing

o   Efficiency, control, choice => Agility

o   Control through policy, service levels, and cost

·         Big  competitors

o   IBM, HP, Cisco, Microsoft, …

o   EMC is big at $20B but not close to as big as these incumbents

o   JRH: I’ve never thought of EMC as the small, nimble competitor but I guess it’s all relative

·         Recent acquisitions in drive to cloud & big data

o   Isilon

o   Greenplum

o   Datadomain

o   RSA

·         Mammoth 5 year M&A plan: roughly ½ of investments in R&D and ½ in M&A

o   14.0B M&A

o   10.5B: R&D

·         EMC has 14,000 sales people so there is huge potential synergy in any acquisition

o   Adding a 14,000 person sales team to any reasonable product is going to produce considerable new revenue quickly

·         EMC is now 152 in fortune 500

o   Revenue is $17B

o   Free cash flow: $3.4b

Thanks to Ed Lazowska for hosting this talk and many in the University of Washington Distinguished Lecturer Series.

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

We see press releases go by all the time and most of them deserve the yawn they get. But, one caught my interest yesterday. At the PASS Summit conference Microsoft Vice President Ted Kummert announced that Microsoft will be offering a big data solution based upon Hadoop as part of SQL Azure. From the Microsoft press release, “Kummert also announced new investments to help customers manage big data, including an Apache Hadoop-based distribution for Windows Server and Windows Azure and a strategic partnership with Hortonworks Inc.”

Clearly this is a major win for the early startup Hortonworks. Hortonworks is a spin out of Yahoo! and includes many of the core contributors to the Apache Hadoop distribution: Hortonwoks Taking Hadoop to Next Level.

This announcement is also a big win for the MapReduce processing model. First invented at Google and published in MapReduce: Simplified Data Processing on Large Clusters. The Apache Hadoop distribution is an open source implementation of MapReduce. Hadoop is incredibly widely used with Yahoo! running more than 40,000 nodes of Hadoop with their biggest single cluster now at 4,500 servers. Facebook runs a 1,100 node cluster and a second 300 node cluster. Linked in runs many clusters including deployments of 1,200, 580, and 120 nodes. See the Hadoop Powered By Page for many more examples.

In the cloud, AWS began offering Elastic MapReduce back in early 2009 and has been expanding the features supported by this offering steadily over the last couple of years adding support for Reserved Instances, Spot Instances, and Cluster Compute instances (on a 10Gb non-oversubscribed network – MapReduces just loves high bandwidth inter-node connectivity)and support for more regions with EMR available in Northern Virginia, Northern California, Ireland, Singapore, and Tokyo.

Microsoft expects to have a pre-production (what they refer to as a "community technology Preview") version of a Hadoop service available by the “end of 2011”.  This is interesting for a variety of reasons. First, its more evidence of the broad acceptance and applicability of the MapReduce model.  What is even more surprising is that Microsoft has decided in this case to base their MapReduce offering upon open source Hadoop rather than the Microsoft internally developed MapReduce service called Cosmos which is used heavily by the Bing search and advertising teams. The What is Dryad blog entry provides a good description of Cosmos and some of the infrastructure build upon the Cosmos core including Dryad, DryadLINQ, and SCOPE.

As surprising as it is to see Microsoft planning to offer MapReduce based upon open source rather than upon the internally developed and heavily used Cosmos platform, it’s even more surprising that they hope to contribute changes back to the open source community saying “Microsoft will work closely with the Hadoop community and propose contributions back to the Apache Software Foundation and the Hadoop project.”  

·         Microsoft Press Release: Microsoft Expands Data Platform

·         Hortonsworks Press Release: Hortonworks to Extend Apache Hadoop to Windows Users

·         Hortonworks Blog Entry: Bringing Apache Hadoop to Windows

Past MapReduce postings on Perspectives:

·         MapReduce in CACM

·         MapReduce: A Minor Step Forward

·         Hadoop Summit 2010

·         Hadoop Summit 2008

·         Hadoop Wins TeraSort

·         Google MapReduce Wins TeraSort

·         HadoopDB: MapReduce over Relational Data

·         Hortonworks Taking Hadoop to Next Level

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

The European Data Center Summit 2011 was held yesterday at SihlCity CinCenter in Zurich. Google Senior VP Urs Hoelzle kicked off the event talking about why data center efficiency was important both economically and socially.  He went on to point out that the oft quoted number that US data centers represent is 2% of total energy consumption is usually mis-understood. The actual data point is that 2% of the US energy budget is spent on IT of which the vast majority is client side systems. This is unsurprising but a super important clarification.  The full breakdown of this data:

·         2% of US power

o   Datacenters:              14%

o   Telecom:                     37%

o   Client Device:            50%

The net is that 14% of 2% or 0.28% of the US power budget is consumed in datacenters.  This is a far smaller but still a very relevant number. In fact, that is the primary motivator behind the conference: how to get the best practices from industry leaders in datacenter efficiency available more broadly .

To help understand why this is important,

·         Of the 0.28% energy consumption by datacenters:

o   Small:            41%

o   Medium:     31%

o   Large:            28%

This later set of statistics predictably shows that the very largest data centers consume 28% of the data center energy budget while small and medium centers consume 72%.  High scale datacenter operates have large staffs of experts focused on increasing energy efficiency but small and medium sized centers can’t afford this overhead at their scale. Urs’s point and the motivation behind the conference is we need to get industry best practices available to all data center operations.

The driving goal behind the conference is that extremely efficient datacenter operations are possible using only broadly understood techniques. No magic is required.  It is true that the very large operators will continue to enjoy even better efficiency but existing industry best practices can easily get even small operators with limited budgets to within a few points of the same efficiency levels.

Using Power Usage Effectiveness as the measure while the industry leaders are at 1.1 to 1.2 where 1.2 means that every watt delivered to the servers requires 1.2 watts to be deliverd from the utility. Effectively it is a measure of the overhead or efficiency of the datacenter infrastructure. Unfortunately the average remains in the 1.8 to 2.0 range and the worst facilities can be as poor as 3.0.

Summarizing: Datacenters consume 0.28% of the annual US energy budget. 72% of these centers are small and medium sized centers that tend towards the lower efficiency levels.

The Datacenter Efficiency conference focused on making cost effective techniques more broadly understood showing how a PUE of 1.5 is available to all without large teams of experts or huge expense. This is good for the environment and less expensive to operate.

                                --jrh

James Hamilton

e: jrh@mvdirona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

I invited Nikhil Handigol to present at Amazon earlier this week. Nikhil is a Phd candidate at Stanford University working with networking legend Nick McKeown on the Software Defined Networking team. Software defined networking is an concept coined by Nick where the research team is separating the networking control plane from the data plane. The goal is a fast and dumb routing engine with the control plane factored out and supporting an open programming platform.

From Nikil’s presentation, we see the control plane hoisted up to a central, replicated network O/S configuring the distributed routing engines in each switch.

One implementation of software defined networking is OpenFlow where each router supports the OpenFlow protocol and a central OpenFlow Controller computes routing tables that are installed in each router:

What makes OpenFlow especially interesting is that it’s simple, easy to implement, and getting broad industry support with the Open Networking Foundation as the central organizing body.  The Open Networking Foundation’s primary mission is to advance software defined networking using OpenFlow as the protocol. Founding members of the Open Networking Foundation are Deutsche Telekom, Facebook, Google, Microsoft, Verizon, and Yahoo!.  Also included are networking equipment providers including: Broadcom, Dell, Cisco, Force10, HP, Juniper, Marvell, Mellanox, and many others.

Today, most networking equipment is shipped as a vertically integrated stack including both the control and data planes. There are many reasons why this is not good for the industry. The Stanford team argues it blocks innovation in that researches can’t try new protocols with a closed stack without a programming model.  I agree. This is a problem for both academia and industry but my dislike of the current model is much broader. In Networking: The Last Bastion of Mainframe Computing, I made the case that this vertically integrated approach is artificially holding prices high and slowing the pace of innovation. A quick summary of the argument:

When networking equipment is purchased, it’s packaged as a single sourced, vertically integrated stack. In contrast, in the commodity server world, starting at the most basic component, CPUs are multi-sourced. We can get CPUs from AMD and Intel. Compatible servers built from either Intel or AMD CPUs are available from HP, Dell, IBM, SGI, ZT Systems, Silicon Mechanics, and many others.  Any of these servers can support both proprietary and open source operating systems. The commodity server world is open and multi-sourced at every layer in the stack.

Open, multi-layer hardware and software stacks encourage innovation and rapidly drive down costs. The server world is clear evidence of what is possible when such an ecosystem emerges.

I’m excited about software defined networking because it provides a clean interface allowing switch providers to both innovate and compete. An additional benefit is that SDN allows innovation and experimentation at the network protocol layer.

In Nikil’s talk last week at Amazon, he explored integrating load balancing functionality into the network routing fabric. The team started with the hypothesis that load balancing is really just smart routing. They then implemented a distributed load balancing fabric by adding load balancing support to network routers using Software Defined Networking. Essentially they distribute the load balancing functionality throughout the network. What’s unusual here is that the ideas could be tested and tried over a 9 campus, North American wide network with only 500 lines of code. With conventional network protocol stacks, this research work would have been impossible in that vendors don’t open up protocol stacks. And, even if they did, it would have been complex and very time consuming.

• Video Demo of the distributed load balancer: http://www.youtube.com/watch?v=Sfqofxdk1gE
• Nikil’s slides from his Amazon Talk: http://www.mvdirona.com/jrh/TalksAndPapers/Handigol-Amazon-May2011.pdf