We see press releases go by all the time and most of them deserve the yawn they get. But, one caught my interest yesterday. At the PASS Summit conference Microsoft Vice President Ted Kummert announced that Microsoft will be offering a big data solution based upon Hadoop as part of SQL Azure. From the Microsoft press release, “Kummert also announced new investments to help customers manage big data, including an Apache Hadoop-based distribution for Windows Server and Windows Azure and a strategic partnership with Hortonworks Inc.”
Clearly this is a major win for the early startup Hortonworks. Hortonworks is a spin out of Yahoo! and includes many of the core contributors to the Apache Hadoop distribution: Hortonwoks Taking Hadoop to Next Level.
This announcement is also a big win for the MapReduce processing model. First invented at Google and published in MapReduce: Simplified Data Processing on Large Clusters. The Apache Hadoop distribution is an open source implementation of MapReduce. Hadoop is incredibly widely used with Yahoo! running more than 40,000 nodes of Hadoop with their biggest single cluster now at 4,500 servers. Facebook runs a 1,100 node cluster and a second 300 node cluster. Linked in runs many clusters including deployments of 1,200, 580, and 120 nodes. See the Hadoop Powered By Page for many more examples.
In the cloud, AWS began offering Elastic MapReduce back in early 2009 and has been expanding the features supported by this offering steadily over the last couple of years adding support for Reserved Instances, Spot Instances, and Cluster Compute instances (on a 10Gb non-oversubscribed network – MapReduces just loves high bandwidth inter-node connectivity)and support for more regions with EMR available in Northern Virginia, Northern California, Ireland, Singapore, and Tokyo.
Microsoft expects to have a pre-production (what they refer to as a “community technology Preview”) version of a Hadoop service available by the “end of 2011”. This is interesting for a variety of reasons. First, its more evidence of the broad acceptance and applicability of the MapReduce model. What is even more surprising is that Microsoft has decided in this case to base their MapReduce offering upon open source Hadoop rather than the Microsoft internally developed MapReduce service called Cosmos which is used heavily by the Bing search and advertising teams. The What is Dryad blog entry provides a good description of Cosmos and some of the infrastructure build upon the Cosmos core including Dryad, DryadLINQ, and SCOPE.
As surprising as it is to see Microsoft planning to offer MapReduce based upon open source rather than upon the internally developed and heavily used Cosmos platform, it’s even more surprising that they hope to contribute changes back to the open source community saying “Microsoft will work closely with the Hadoop community and propose contributions back to the Apache Software Foundation and the Hadoop project.”
· Microsoft Press Release: Microsoft Expands Data Platform
· Hortonsworks Press Release: Hortonworks to Extend Apache Hadoop to Windows Users
· Hortonworks Blog Entry: Bringing Apache Hadoop to Windows
Past MapReduce postings on Perspectives:
· MapReduce: A Minor Step Forward
· Google MapReduce Wins TeraSort
· HadoopDB: MapReduce over Relational Data
· Hortonworks Taking Hadoop to Next Level
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
Perhaps you are right that Microsoft is using this as a form of a hedge rather than a primary strategy. Probably true given how aggressive the Microsoft legal team has been against users of open source software.
Nonetheless, this particular announcement looks like a good thing for the industry and its good to see.
–jrh
It seems like Microsoft is testing a new strategy of playing nice with certain popular and potentially disruptive open source projects and making sure they have official and performant Windows support. The FOSS community largely isn’t interested in providing first-class Windows support and they are losing serious mindshare with the next generation — which over time is going to eat into the Server business. Node.js is another example in recent months, where MS is working with Joyent to create a (rather nifty) cross platform I/O eventing library in libuv, which is core to Node. OpenNebula is another example. I would not be surprised to see more of this kind of thing as a hedge.