Tuesday, July 16, 2013

At the Microsoft World-Wide Partners Conference, Microsoft CEO Steve Ballmer announced that “We have something over a million servers in our data center infrastructure. Google is bigger than we are. Amazon is a little bit smaller. You get Yahoo! and Facebook, and then everybody else is 100,000 units probably or less.

 

That’s a surprising data point for a variety of reasons. The most surprising is that the data point was released at all. Just about nobody at the top of the server world chooses to boast with the server count data point. Partly because it’s not all that useful a number but mostly because a single data point is open to a lot of misinterpretation by even skilled industry observers. Basically, it’s pretty hard to see the value of talking about server counts and it is very easy to see the many negative implications that follow from such a number

 

The first question when thinking about this number is where does the comparative data actually come from?  I know for sure that Amazon has never released server count data. Google hasn’t either although estimates of their server footprint abound. Interestingly the estimates of Google server counts 5 years ago was 1,000,000 servers whereas current estimates have them only in the 900k to 1m range.

 

The Microsoft number is surprising when compared against past external estimates, data center build rates, or ramp rates from previous hints and estimates. But, as I said, little data has been released by any of the large players and what’s out there is typically nothing more than speculation. Counting servers is hard and credibly comparing server counts is close to impossible.

 

Assume that each server runs 150 to 300W including all server components with a weighted average of say 250W/server. And as a Power Usage Effectiveness estimator, we will use 1.2 (only 16.7% of the power is lost to datacenter cooling and power distribution losses). With these scaling points, the implied total power consumption is over 300MW. Three hundred million watts or, looking at annual MW-h, we get an annual consumption of over 2,629,743 MWh or 2.6 terawatt hours – that’s a hefty slice of power even by my standards. As a point of scale since these are big numbers, the US Energy Information Administration reports that in 2011 the average US household consumed 11.28MWh. Using that data point, 2.6TWh is just a bit more than the power consumed by 230,000 US homes.

 

Continuing through the data and thinking through what follows from “over 1M” servers, the capital expense of servers will be $1.45 billion dollars assuming a very inexpensive $1,450 per server. Assuming a mix of different servers with an average cost of $2,000/server, the overall capital expense would be $2B before looking at the data center infrastructure and networking costs required to house them. With the overall power consumption computed above of 300MW which is 250MW of critical power (using a PUE of 1.2) and assuming a data center build cost at a low $9/W of critical load (Uptime Institute estimates numbers closer to $12/W), we would have a data center build cost of $2.25B dollars.  The implication of “over 1M servers” is an infrastructure investment of $4.25B including the servers. That’s an athletic expense even for Microsoft but certainly possible.

 

How many datacenters would be implied by “more than one million servers?” Ignoring the small points of presence since they don’t move the needle, and focusing on the big centers, let’s assume 50,000 servers in each facility. That assumption would lead to 30 major facilities.  As a cross check, if we instead focus on power consumption as a way to compute facility count and assume a total datacenter power consumption of 20MW each and the previously computed 300MW total power consumption, we would have roughly 15 large facilities. Not an unreasonable number in this context.

 

The summary following from these data and the “over 1M servers” number:

·         Facilities: 15 to 30 large facilities

·         Capital expense: $4.25B

·         Total power: 300MW

·         Power Consumption: 2.6TWh annually

 

Over one million servers is a pretty big number even in a web scaled world.

 

                                                --jrh

 

Data Center Knowledge article: Ballmer: Microsoft has 1 Million Servers

Transcript of Ballmer presentation:  Steve Ballmer: World-Wide Partner Conference Keynote


Additional note: Todd Hoff of High Scalability made a fun observation in his post Steve Ballmer says Microsoft has over 1 million servers. From Todd: “The [Microsoft data center] power consumption is about the same as used by Nicaragua and the capital cost is about a third of what Americans spent on video games in 2012.


 James Hamilton 

e: jrh@mvdirona.com 
w: 
http://www.mvdirona.com 
b: 
http://blog.mvdirona.com / http://perspectives.mvdirona.com

 

 

Tuesday, July 16, 2013 4:34:17 PM (Pacific Standard Time, UTC-08:00)  #    Comments [11] - Trackback
Services
Tuesday, July 16, 2013 7:43:14 PM (Pacific Standard Time, UTC-08:00)
as always great article. You could power quite a few countries off that. Thinking about this also how much is it costing in the Opex electric, water and other utilities. plus the carbon output.
Tuesday, July 16, 2013 8:09:06 PM (Pacific Standard Time, UTC-08:00)
They leave themselves open to the interpretation "Windows is so much less efficient than the alternatives that we have to have way more servers despite having fewer customers and activity". The numbers they should brag about are ratios - servers per system ops/admin person, annual power consumption per user, servers per user etc.
Roger
Wednesday, July 17, 2013 1:21:36 AM (Pacific Standard Time, UTC-08:00)
Why is it so important for a company to keep the number of its servers in secret ?
Dimitris
Friday, July 19, 2013 2:00:37 AM (Pacific Standard Time, UTC-08:00)
Pretty sure a significant number of those "servers" are Virtual Machines, it doesn't necessarily mean 1M physical, tin servers @250W/each.

More like 10 - 20 x less (10-20 VMs per physical host).
Friday, July 19, 2013 10:36:29 AM (Pacific Standard Time, UTC-08:00)
Dmitris asked "Why is it so important for a company to keep the number of its servers in secret ?"

I don't know that it is important to keep them secret. In fact, these data points are super interesting to a wide range of consumers. The industry as a whole is interested in scale right now so the popular press covers it. Financial analysis are interested in tracking expenses and how they line up against revenue streams, so these data are interesting to them as well. Environmentalist care about power consumption, so these data are interesting to them as well. There are lots of folks interested in this information so releasing it is probably a good thing.

The comparison made by Steve Ballmer to Google and Amazon server front prints is pure speculation rather than information so I would have preferred that it have been omitted. Speculation in the same sentence as a useful data point, draws the accuracy of the data point itself into question.

--jrh
Friday, July 19, 2013 10:40:34 AM (Pacific Standard Time, UTC-08:00)
Simon suggests "Pretty sure a significant number of those "servers" are Virtual Machines."

Certainly it is possible that the Microsoft server count is virtual but Ballmer did fairly succinctly say "We have something over a million servers in our datacenter infrastructure." Counting virtual servers would be a pretty misleading. Possible but I hope not and, if they really are virtual, it's a very small number.

--jrh
Tuesday, July 30, 2013 10:49:11 AM (Pacific Standard Time, UTC-08:00)
Thanks for the insight. Trying to reproduce some of the math. 250MW x $9/MW = $2250 as opposed to $2.25B. What am I doing wrong. Thanks again.
Mark
Thursday, August 01, 2013 11:22:08 AM (Pacific Standard Time, UTC-08:00)
Good catch Mark. The total is correct but the line item should read $9/W rather than $9/MW. Thanks,

--jrh
Tuesday, August 06, 2013 7:36:03 PM (Pacific Standard Time, UTC-08:00)
Greenpeace estimated the max. power demand of several companies' datacenters a few years ago (link below). Your estimate for Microsoft is not far off theirs (252 MW).

http://www.greenpeace.org/international/en/publications/Campaign-reports/Climate-Reports/How-Clean-is-Your-Cloud/; annex contains details on each datacenter
Mark Ellis
Tuesday, August 06, 2013 7:51:53 PM (Pacific Standard Time, UTC-08:00)
Yes,there certainly are no shortage of server count and power consumption estimates out there. But, except for rare cases like this where a company releases data on their own fleet, the estimates are all pretty thin on data.

--jrh
Friday, August 16, 2013 8:24:00 PM (Pacific Standard Time, UTC-08:00)
I'm not sure if you've seen this. Ebay has started publishing their heater scale (power dissipation), PUE, WUE, server counts, etc. Not huge as an org in scale in comparison to GOOG|MSFT|AMZN, but interesting.

Ebay server count blog article



PS: I would absolutely *shocked* if VMs exist to any statistically significant degree, in regards to other comments on composition.
Sam
Comments are closed.

Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.

Archive
<July 2014>
SunMonTueWedThuFriSat
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789

Categories
This Blog
Member Login
All Content © 2014, James Hamilton
Theme created by Christoph De Baene / Modified 2007.10.28 by James Hamilton