Facebook Iowa Data Center
In 2007, the EPA released a study on datacenter power consumption at the request of the US Congress (EPA Report to Congress on Server and Data Center Efficiency). The report estimated that the power consumption of datacenters represented about 1.5% of the US Energy Budget in 2005 and this number would double by 2010. In a way, this report was believable in that datacenter usage was clearly on the increase. What the report didn’t predict was the pace of innovation in datacenter efficiency during that period. Increased use, spurred increased investment, which has led to a near 50% improvement in industry leading datacenter efficiency.
Also difficult to predict at the time of the report was the rapid growth of cloud computing. Cloud computing is a fundamentally more efficiently way to operate compute infrastructure. The increases in efficiency driven by the cloud are many but a strong primary driver is increased utilization. All companies have to provision their compute infrastructure for peak usage. But, they only monetize the actual usage which goes up and down over time. What this leads to incredibly low average utilization levels with 30% being extraordinarily high and 10 to 20% the norm. Cloud computing gets an easy win on this dimension. When non-correlated workloads from a diverse set of customers are hosted in the cloud, the peak to average flattens dramatically. Immediately effective utilization sky rockets. Where 70 to 80% of the resources are usually wasted, this number climbs rapidly with scale in cloud computing services flattening the peak to average ratio.
To further increase the efficiency of the cloud, Amazon Web Services added an interesting innovation where they sell the remaining capacity not fully consumed by this natural flattening of the peak to average. These troughs are sold on a spot market and customers are often able to buy computing at less the amortized cost of the equipment they are using (Amazon EC2 Spot Instances). Customers get clear benefit. And, it turns out, it’s profitable to sell unused capacity at any price over the marginal cost of power. This means the provider gets clear benefit as well. And, with higher utilization, the environment gets clear benefit as well.
It’s easy to see why the EPA datacenter power predictions were incorrect and, in fact, the most recent data shows just how far off the original estimates actually were. The EPA report estimated that in 2005, the datacenter industry was consuming just 1.5% of the nation’s energy budget but what really caused international concern was the projection that this percentage would surge to 3.0% in 2010. This 3% in 2010 number has been repeated so many times that it’s now considered a fact by many and it has to have been one of the most referenced statistics in the datacenter world.
The data on 2010 is now available and the power consumption of datacenters in 2010 is currently estimated to have been in the 1.1 to 1.5% in 2010. Almost no changes since 2005. Usage has skyrocketed. Most photos are ending up in the cloud, social networking is incredibly heavily used, computational science has exploded, machine learning is tackling business problems previously not cost effective to address, much of the world now have portable devices and a rapidly increasing percentage of these devices are cloud connected. Server side computing is exploding and yet power consumption is not.
However, the report caused real concern both in the popular press and across the industry and there have been many efforts to not only increase efficiency but to also to increase the usage of clean power. The progress on efficiency has been stellar but the early efforts on green power have been less exciting.
I’ve mentioned some of the efforts to go after cleaner data center power in the past. The solar array at the Facebook Prineville datacenter is a good example: It’s a 100kW solar array “powering” a 25MW+ facility. Pure optics with little contribution to the datacenter power mix (I love solar but… ). Higher scale efforts such as the Apple Maiden North Carolina facility are getting much closer to being able to power the entire facility go far beyond mere marketing programs. But, for me, the land consumed is disappointing with 171 acres of land cleared of trees in order to provide green power. There is no question that solar can augment the power at major datacenters but onsite generation with solar just doesn’t look practical for most facilities particularly those in densely packed urban settings. In the article Solar at Scale: How Big is a Solar Array with 9MW Average Output, we see that a 9MW average output solar plant takes approximately 314 acres (13.6 million sq/ft). This power density is not practical for onsite generation at most datacenters and may not be the best approach to clean power for datacenters.
The biggest issue with solar applied to datacenters is the mass space requirements are not practical with most datacenters being located in urban centers and it may not even be the right choice for rural centers. Another approach gaining rapid popularity is the use of fuel cells to power datacenters. At least partly because of highly skilled marketing, fuel cells are considered by many jurisdictions to be “green”. The good news is that some actually are green. A fuel cell running on waste biogas is indeed a green solution with considerable upside. This is what Apple plans for its Maiden North Carolina fuel cell farm and it’s a very nice solution (Apple’s Biogas Fuel Cell Plant Could Go Live by June).
The challenge with fuel cells in the datacenter application is just about all of them are not biogas powered relying instead on non-renewable energy. For example, the widely heralded eBay Salt Lake City fuel cell deployment is natural gas powered (New eBay Data Center Runs Almost Entirely on Bloom Fuel Cells). There have been many criticisms of the efficiency of these fuel cell-based power plants (884 may be Bloom Energy’s Fatal Number and their cost effectiveness (Delaware Press Wakes Up to Bloom Energy Boondoggle).
I’m mostly ignoring these reports and focusing on the simpler question: how can running a low-scale power plant on a fossil fuel possibly be green and is on-site generation really going to be the most environmentally conscious means of powering data centers? Most datacenters lack space for large on-site generation facilities, most datacenter operators don’t have the skill, and generally the most efficient power generation plants are far bigger than a single datacenter could ever consume on its worst day.
The argument for on-site generation is to avoid the power lost in transmission. The US Energy Information Administration (EIA) estimates power lost in transmission to be 7%. That’s certainly a non-zero number and the loss is relevant but I still find myself skeptical that the world leaders in operating efficient and reliable power plans are going to be datacenter operators. And, even if that prospect does excite you, does having a single plant on one location being the sole power source for the datacenter sounds like a good solution? I’m finding it hard to get excited. Maintenance, natural disasters, and human error will eventually yield some unintended consequences.
A solution that I really like is the Facebook Altoona Iowa facility (Facebook Says Its New Data Center Will Run Entirely on Wind). It is a standard grid connected datacenter without on-site power generation. But they have partnered to have built a 138MW wind project in nearby Wellsburg Iowa. What I like about this approach is: 1) no clear cutting was required to prepare the land for generation and the land remains multi-use, 2) it’s not fossil fuel powered, 3) the facility will be run by a major power generation operator rather than as a sideline by the datacenter operator, and 4) far more clean power is being produced than will be actually used by the datacenter so they are actually adding more clean power to the grid than they are consuming by a fairly significant margin.
Hats off to Facebook for doing it clean data center energy right. Nice work.
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
I agree. The report missed the mark in fully discribing how power is consumed in IT infrastrucure and where the bulk of it is consumed. And, in hindsight, we see the estimate of growth was incorrect as well.
It’s an important topic but its more complex topic than "IT power consumption is bad" and it is deffinitely not the case that all IT power consumption looks the same.
Thanks James. At the time of its publication, the 2007 EPA report was in my view well-intentioned and already long overdue. It attained seminal status at once, as evidenced by extraordinary number of times and places it’s since been referenced and inserted into a broad range of industry sectors and applications. As such it became a "foundational" reference with all the weight of canon used for all manner of estimates and projections across diverse paradigms. For example, even federal and state recommendations for Building Energy Benchmarking reporting requirements result in HVAC (especially the proportional amount of HVAC supply going to IT from chiller plants) appear on the "Base Building" ledger, as opposed to "Tenant Systems", thus removing the burden from IT to optimize usage. The negative implications relating to energy efficiency tax incentives and rebate programs are absolutely huge, beyond the scope of this writing, since these factors are only vaguely understood, if at all, by utilities or regulators. Suffice, unfortunately the original findings were nebulous and in many cases uninformed, which has led to endless distortions and nonsense as we attempt to optimize energy usage. I trust these issues will resolve themselves before singularity sets in ;)
Frank
Frank brings up several points of which two are well worth underlining: 1) most compute power is in the lower end of the consumption pyramid mostly because there are SOOOOO many of them (portable device counts make servers counts look like rounding error and, moving up a tier, networking rooms, laptops, and in-office server room deployments vastly dwarf most companies central IT consumption), and 2) most networking and server closets power and cooling is not visible to the folks running IT and is just about never reported even though these faciliites offten consume more power than central IT. The bottom of the pyramid is where all the use is and, even though the power/device is usually low, the aggregate consumption is not.
It’s a great point that this consumption is under reported and deserves more focus. Many of these deployements are incredibly inefficient and, alhtough small individually, there are thousands of them.
–jrh
James,
Aside from your ongoing and excellent coverage of green data centers and the factors affecting the use of renewables for same, I’d like to address with you and your readers a nit that I’ve had with the EPA’s 1.5% statement since it was first published some six years ago. Maybe more to the point, I don’t understand it and have not been able to reconcile several apparent disconnects that I find with its methodologies and results, despite having read the report several times. There seems to be a dichotomy of factors and criteria at work here, a sort of IT schizophrenia with respect to the types of asset classes and accounting rules applied to IT energy consumption measurements from one type of facility to another.
[It’s alarming to hear distortions coming from presumably knowledgeable enterprise IT and Facilities management types claiming that their office building’s total IT energy burn amounts to only 1.5% of the building’s total electric bill!]
For example, why is it that A/C and UPS found in the larger data centers’ are included as part of the whole (the PUE numerator), but when it comes to enterprise office buildings and campuses (which collectively contain millions of equipment rooms, effectively mini-data centers akin to modular/container type DCs housing LAN equipment and residual servers), the air conditioning and UPS systems found in those smaller rooms are excluded entirely. Worse, no one as far as I’m able to discern is even measuring PUE in the latter venues, which is odd when you consider the amount of burn taking place there. Are the latter not also considered a part of IT? If so, why are they being assessed differently?
That said, as was the case in the data center until recently, these latter assets consisting of HVAC and UPS situated in smaller equipment rooms have not generally been considered a part of IT’s responsibility, but instead have traditionally fallen to the facility management silo to administer and pay for, just as it always did for decades. This has been a cultural constant that is only now beginning to change, albeit slowly, in larger data centers first. Why is it not also changing in office buildings and campus settings at the same pace? These observations may appear superficial or even trivial, until one realizes that the total energy burn in those smaller rooms (which also employ HVAC and UPS) exceeds the total burn rate of their larger data center kin. Or at least that is what I’m led to infer. But estimates of the total number and types of data centers has also been erratic, to say the least, ranging from 50,000 to 2.9 million <!>, so I’ll be less that absolute about any of this. And that, too, is a factor which causes doubts as to the accuracy of the percentages cited.
All of which brings me back to my earlier point concerning the EPA’s initial and oft’ repeated findings of then 1.5% and now 3.0%. You stated it’s as though the 3.0% figure is now being regarded as fact? I’ve always maintained a similar view about the legitimacy of the original estimate of 1.5%, that it too was incorrect and being regarded as fact.
How many of the contributory factors affecting PUE cited above were included in the calculus back in 2007, and which ones might have been arbitrarily excluded, both by asset classification and venue? I am interested in any insights you and your readers may have concerning these apparent disconnects. To all, TIA!
frank@cirrant.com
Dave asked "Are we thinking too rigidly? we have tons of non-tree populated square area".
That’s true on a country-wide bases but not, in general, true of data centers. Most are housed in urban areas where large arrays are impractical. Those in rural areas could deploy arrays but I’m arguing the data center sites are not the best places to put power plants, data center operators are not the best folks to operate power plants, and solar power density and production schedules are not well suited to data center power demands.
Are we thinking too rigidly? we have tons of non-tree populated square area. http://www.solarroadways.com seems to address this. Price appropriate right now? no. But given the amount we as a countrry pour into the oil and gas markets to subsidize our energy there in addition to the amount we spend aggressively securing supply lines… it becomes more reasonable to start thinking through how all of the electric car batteries can have a second life when they reach a point where they are yanked from cars due to lack of range but still have plenty of utility grade peak power storage for electric/wind.
just saying – thinking outside the box of building super huge arrays that use up a ton of new space…