I’m Interested in data center resource consumption in general and power is a significant component of overall operating cost and also has impact on the environment so, naturally, it gets most of the focus when discussing data center resource consumption. As with all real issues, there is always a bit of hyperbole and some outright confusion in some of the reports focused on resource consumption. In many cases I’ve spoken with the authors, at times I’ve written about it more broadly (Greenpeace, Renewable Energy, and Data centers). David Linthicum of Infoworld has made similar points in his recent article: Is Your Cloud Green Enough: That’s the wrong question.
In a widely sighted 2007 article EPA Report to Congress on Server and Data Center Efficiency, the EPA estimated that data centers were consuming 1.5% of the 2005 US Energy Budget and, what was really troubling was, this number was expected to double over the subsequent 10 years. This report generated a fair amount of excitement and interest at the time and it’s subsequently been quoted so many times that it is still largely accepted as fact.
It’s now been 10 years so we can just go back and check on the original EPA prognostication. The growth of data center usage that they predicted appears to have happened at roughly the scale they estimated. But aggregate data center power consumption is still about the same as it was 10 years ago when they first looked at the issue. The progress on increasing server and overall data center efficiency has very close to zeroed out the expected increase in power consumption. From an earlier blog post Data Center Renewable Power Done Right:
The data on 2010 is now available and the power consumption of datacenters in 2010 is currently estimated to have been in the 1.1 to 1.5% in 2010. Almost no changes since 2005. Usage has skyrocketed. Most photos are ending up in the cloud, social networking is incredibly heavily used, computational science has exploded, machine learning is tackling business problems previously not cost effective to address, much of the world now have portable devices and a rapidly increasing percentage of these devices are cloud connected. Server side computing is exploding and yet power consumption is not.
It’s a pretty good story of technology successfully battling an industry problem. From 50 Years of Moore’s Law: IEEE Spectrum Report, there is an interesting article Moore’s Law Might be Slowing Down, but not Energy Efficiency where the authors report:
Encouragingly, typical-use efficiency seems to be going strong, based on tests performed since 2008 on Advanced Micro Devices’ chip line. Through 2020, by our calculations for an AMD initiative, typical-use efficiency will double every 1.5 years or so, putting it back to the same rate seen during the heyday of Moore’s Law. These gains come from aggressive improvements to circuit design, component integration, and software, as well as power-management schemes that put unused circuits into low-power states whenever possible. The integration of specialized accelerators, such as graphics processing units and signal processors that can perform certain computations more efficiently, has also helped keep average power consumption down.
In my view, we are just getting started and easily have another 10 years of fast paced innovation in front of us.
In 2009 in Data Center Efficiency Best Practices, I said it’s “not just about power” and argued that water consumption is the next big natural resource issue for data centers after power. I think it’s still true today and expect that water consumption will continue to need to be managed carefully by all data center operators. A recent Wall Street Journal article written Drew Fitzgerald Data Centers and Hidden Water Use recently came out referencing this data point and talking about the importance of data center water conservation especially in light of the current water shortage in California.
Taking a mathematical perspective, the water problem is actually reducible to the power problem since power and plentiful sea water can be turned into drinking water using techniques such as reverse osmosis (R/0) on polluted water or even sea water. In fact R/O plants can be found at some data centers today used in many applications from reducing the mineral content in blow down water leaving the data center (water discarded from the water tower condensate loop) through to purifying cooling water entering the facility for use in spray cooling systems (Open Compute Mechanical System Design). I even have an R/O system on my boat to produce drinking water from sea water. They are certainly handy devices but R/O systems aren’t magic. They are fairly expensive and power intensive so the water issue remains one worth working upon. After power, it’s still the most important environmental resource consumption issue facing the industry.
Just as we can actually create drinking water from either polluted water or sea water, it’s also possible to completely eliminate the use of water in data center cooling. The reason this isn’t already done broadly is that the commonly used water-free cooling systems are far less power efficient. So water-free data center operation doesn’t look like the best environmental choice.
Two examples that I really like of innovative ways to reduce or eliminate water consumption in data centers without excessive power consumption are the Google St. Ghislain Belgium facility that is cooled using waste water that is first purified on site and the Google Hamina Finland center that is cooled using salt water. It is of course worth mentioning that one of the best ways to consume less water in cooling data centers is to cool them less and, over the last 10 years, this has approach has had a huge impact on overall consumption that has not yet rippled through the entire industry.
Just as was the case with power, it’s easy to see big water consumption numbers and over-react. This is an area where the industry will keep innovating and I expect to see results along the line as we have seen with power where compute usage has gone up dramatically but power consumption has not.
Mark Andreesen is good at seeing past general beliefs and frequently referenced statistics and offering a different and oftentimes more thoughtful perspective. In looking at the data center water consumption issue, Adreesen starts by noting that the water consumption of California’s 800 data centers is a big number but then points out that, in the US alone, newsprint production consumes a prodigious 109,500,000,000 gallons per year. He then argues that the move to online news consumption is actually reducing water consumption. The original Andreesen tweet storm:
Marc Andreessen @pmarca 5h5 hours ago
1/Interesting analysis in @WSJ of the current water use of California’s 800 commercial computer datacenters:http://www.wsj.com/articles/SB10007111583511843695404581067903126039290 …
Marc Andreessen @pmarca 5h5 hours ago
2/California’s datacenters consume 158,000 Olympic sized swimming pools of water per year, or 104,280,000,000 gallons of water per yaer.
Marc Andreessen @pmarca 5h5 hours ago
3/That sounds like a lot. Enough, in fact, that one might wonder what else might consume that much water every year.
Marc Andreessen @pmarca 5h5 hours ago
4/It turns out that a single day’s supply of newspaper newsprint requires about 300,000,000 gallons of water.https://www.nycwatershed.org/pdfs/WaterTriviaQuestions.pdf …
Marc Andreessen @pmarca 5h5 hours ago
5/Multiply by 365 days per year, producing the US’s annual consumption of newsprint consumes 109,500,000,000 gallons of water per year.
Marc Andreessen @pmarca 5h5 hours ago
6/109,500,000,000 gallons per year for US newsprint > 104,280,000,000 gallons for California datacenters. Interesting!
Marc Andreessen @pmarca 5h5 hours ago
7/Of course newsprint consumption is falling due to substitution by online media! And we know online news is a very small % of online use.
Marc Andreessen @pmarca 5h5 hours ago
8/So, for a small fraction of the water use by datacenters, over time we can likely eliminate nearly all of the water use for newsprint.
Marc Andreessen @pmarca 5h5 hours ago
9/Not to mention reducing the deforestation caused by newsprint production. Less tree use, less water use = environmental win/win!
Marc Andreessen @pmarca 5h5 hours ago
10/It’s interesting to consider what other current environmental resource uses will be reduced or eliminated by datacenters & online media!
As always, an interesting perspective. The industry will keep working on power and water conservation, utilization, and sourcing and I expect another decade of innovation to be coming but it’s good to get a reminder once in a while on how Data Center consumption relates to the rest of the world.
James, this is really helpful reading about “Data Center Power & Water Consumption”! Great article thanks and keep posting things like this.
Yeah, I should get another post up soon. Thanks for the reminder Neil.
9.26.2017
With rise in IT spending and inclusion of IT in every aspect and operations has given to generation of huge amounts of data. Such data are stored, processed, transferred through a large number of servers, which are stacked. Day in day out these servers along with potential cyber threats have one primary threat which is heat. Every electronic devices generates heat as they run on power source either A.C or in D.C. For proper functioning of such servers and IT systems heat generated need to be controlled in order to prevent any malfunctioning, melting and damages of server’s chipsets and components. Cooling of computer components with liquids originated in the 1970s with IBM 3033 and the Cray-2. In the last ten years, however, with the increased awareness and initiative to go “green” to reduce energy consumption, developing viable industrial-grade liquid cooling systems for data center use became a priority. Current technology employs liquid immersion systems: submerging servers and other components in thermally, but not electrically, conductive liquids such as mineral-based oils. Over the forecast period, it is anticipated data center liquid cooling market is expected to grow over the forecast period. It also seems that the main usable waste bi-product is heat waste that could be processed,harnessed & directed as supplemental heating during cooler months when a foot print design places these server units underground beneath large office,retail & habitable building spaces. Do you project seeing these initiatives enhancing cooler northern climates??? Perhaps the underlying question no one wants to hear- do we really need such toys in our back yards? Back in the day talking to a person on the phone or listening to someone on the radio seemed more rewarding!
You are right that there are good reasons to place data centers in cool locations. There are also advantages to placing data centers were property is inexpensive. Stabe governments are a requirement. Low cost power is helpful. Near to renewable energy source is a strong attraction as well. But, there are some factors that force some important workloads in areas that appear less than ideal. In the the article “Why are there so many data centers in New York, Hong Kong, and Tokyo” I walk through some of the reasons why data centers are sometimes placed where land is expensive, solar and wind power is impractical, and power is expensive:
//perspectives.mvdirona.com/2012/07/why-there-are-datacenters-in-ny-hong-kong-and-tokyo/
9.25.2017
Oracle isn’t leaving any stone unturned in its efforts to scale up in the cloud space. Although Oracle is scaling up rapidly, it lags behind Amazon (AMZN), Microsoft (MSFT), and Alphabet (GOOG) in the cloud infrastructure space. This inability of competitors to work together as a team toward one smooth common goal will only further lead to frictional consumer confusion and sporadic blind spending. Where is the efficiency in that??
9.23.2017
Hi
Whats your take on north & south poll data center cooling efforts?? Hint ice!
Just as we can actually create drinking water from either polluted water or sea water, it’s also possible to completely eliminate the use of water in data center cooling. The reason this isn’t already done broadly is that the commonly used water-free cooling systems are far less power efficient. So water-free data center operation doesn’t look like the best environmental choice.
Two examples that I really like of innovative ways to reduce or eliminate water consumption in data centers without excessive power consumption are the Google St. Ghislain Belgium facility that is cooled using waste water that is first purified on site and the Google Hamina Finland center that is cooled using salt water. It is of course worth mentioning that one of the best ways to consume less water in cooling data centers is to cool them less and, over the last 10 years, this has approach has had a huge impact on overall consumption that has not yet rippled through the entire industry.
That’s correct: 100% water free data center operation is easy to do build but these designs are considerably less power efficient so not great environmental choices. The interesting challenge is efficient use of water and then using non-potable water sources like the two examples you pointed out.
http://en.community.dell.com/dell-blogs/dell4enterprise/b/dell4enterprise/archive/2016/06/02/the-unveiling-of-a-revolutionary-new-liquid-cooling-technology
James, wondering your thoughts……
I do work for Dell but have nothing to do with this product….
Water cooling will return and in some workloads it has already returned. Towards the end of the Emitter Coupled Logic (ECL) era of mainframes before they went to CMOS, IBM was using water cooling and needed it at the time. CMOS is so much more efficient that air cooling was sufficient and cheaper overall. As power densities continue to increase and we continue to push CMOS power densities hard, we will eventually need to return to water cooling. Some HPC workloads are already more effectively cooled in water.
The reason not all workloads have already returned to water is partly friction — changes happens slowly and water on the floor of a data center makes operators nervous. But it’s partly the effectiveness of modern air cooled designs. Old air cooled designs stretch further than anyone would have expected. Consequently, most systems and workloads are still most efficiently cooled by air in the models I use. I’m super interested in water because the end is near and there will be more and more designs that need water to fully exploit the design.
The downside of water is it expensive. Of course, heat sinks and moving air and possibily chilling air is expensive as well. Depending upon how efficient a given operator is at moving air and how good their server designs are, they can put off water cooling for longer and still have a less expensive overall solution. But, even for the best operators using the most advanced designs, it’s comming.
There are some very good resources about immerse computers on oil…
http://info.grcooling.com/data-center-energy-efficiency
Good resource and I am seeing some customers in the HPC market where power densities have been highest. At the highest power densities, it is a win. The article you reference argues it’s a win at all power densities but my models don’t agree with that more general point. I do agree it’s a great solution at the highest densities and these densities will one day be the norm.
Water is our biggest problem, as we take it for granted. You still drive across the USA and see how we use drinking water to flush our toilets, water our gardens,etc . It is just unbelievable
The whole subject of water consumption is unavoidable in any industry ( pulp & paper, oil , power,etc). With uses ranging from cooling to process its very difficult to live without it in our businesses. Just using other industries to state they use more water than data centers is useless. Moving publishing to online and just stating that this will have an effect on water use is a huge leap and I think we will experience a bit of Jevons paradox.
I would not use the term of ” problem or waste” within our data centers.
We have to look at the industry that now has matured enough to understand we have and tackle our water management within mission critical facilities.
In order for us to declare waste or problem we have to proof this against a benchmark,we have to validate against the best available technology at the time when we design these facilities and of course evaluate the “value” of the overall approach.
We tackle the power problem by some simple high level steps:
a) Understanding the ratios of efficiencies versus work gain , roughly PUE
b) Clearly enabling our teams to understand every element of the power train and elements to tweak to extract overall efficiency gains by reducing our PUE value or goals
b) Having a power industry that has rates and tariffs that are structured the same way across the nation that at least allow us to price the use of power from the entrance all the way to the rack
With water the issues are a bit different:
a) We have to create some metric, the green grid had one a while back. But it is not as straight forward
b) The water train needs to be developed and we need to be able to understand the different levers and associated impacts. Psychometric charts, environmental envelopes, SLA’s have big impact on these
c) I think the biggest issue is understanding “the water bill”. The whole water industry is disjointed, the rates structures are a mess, how the price this element is just pure madness.
d) But also we need to understand our power source as this consumes water and we should include it as part of the data center water efficiency model. Natural gas plants in states like California, Utah,etc are huge water hoggers as they need to cool the inlet of the turbines to improve the efficiency points
In any case James thanks for raising the issue further. BTW I am enjoying your trip blog
Thanks for the comment David. I have no direct debate with your points but it’s important for us to keep two points in mind: 1) we can design and build data centers today that consume no water (at a slight increase in power consumption), and 2) we can create potable water from sea water with power. The first points tells us we have a way out if necessary and the second point says, at a cost, we can actually create potable water so we have a solutio in the limit. Essentially, this is the number 2 resource issue after power and considerable innovation is needed but we are in control and have levers we can pull as an industry to mitigate consumption as needed.
Agree, with your statement James we can build the solution. What has hold up the wide adoption is that everyone has a different “value” assigned to the solutions and are focused more on power than on water because the impact is not quite clear.
When the PUE system was adopted everyone started to compete to reduce the number for purposes of :
a) Lower PUE implied potentially lower power costs and also for advanced clients a “good citizen” statement
b) PUE advertisement translated in good news, good competition to wholesale and colo providers,etc
c) Give us a simple benchmark to measure and our improvement will be accelerated
Desalinization technologies are still costly, difficult from the standpoint of environmental approvals and are huge power consumers. This is a very big topic in oil and power industry
Recycling of water at a municipal level needs to accelerate, maybe data centers owners can take owners of taking tax incentives/rebates and use part of this to develop some of these facilities in states with limited water resources
This is a very difficult topic but requires all of the attention possible
David, it is true that there is no common “cost” for water nor widely adopted metrics for tracking water consumption per data center size. This could help.
Some juristictions do require that the blow down water from cooling towers be treated so you will find facilities that do recycle. Again, it can be done and the technologies are widely understood. More innovation and more use will drive down costs but, generally, the tools are available.
9.23.2017
When units are to be cooled quickly & more efficiently place smaller units in naturally cooler air/water environments. It doesn’t take any additional energy to produce naturally occurring: ocean currents,wind power,air temps.water falls to mention a few. We humans have designed & built water dams,laid vast amounts of cable across oceans,built water powered saw mills,wind mills the list goes on. Certainly we have the ability to design environmentally approved efficient solid state of the art units!
The question is-can we as humans survive without our natural resources! Not for long.
:
Sadly the water does not return to the system. Current generation economizers rely on evaporation. We did institute a water monitoring dashboard (http://www.opencompute.org/blog/a-new-way-to-report-pue-and-wue/) and James points out a number of co-generation possibilities, but they all make site selection even more challenging.
Much work is going into finding other materials to select for phase-change heat transfer, mixed gasses in tubes is seeing a resurgence, and some salt ‘evaporation’ techniques are able to do recovery, but are not at scale as yet. If we had the right to ask for anything at all, it would be to have the heat contained in such a way that only the components needing cooling would have the cooling delivered. Meaning if we could run a compute blade at 40C inlet, and had a way to cool just the memory and CPU, say by gas-pipe to the sidewall, then we would be able to move a much smaller amount of air, and potentially reduce/remove water from the equation (except in CA where 30% of our power is hydro electric..).
James and Mark are quite right, this issue is not going to solve itself, it may mean next-gen data centers are laid out differently, with a focus not on blowing air past hot-spots, but moving hot-spots to places where it is easier to remove the heat.
Is it really meaningful to talk about water “use” in the context of data centers? It isn’t as if the water disappears forever as is consumed. Data centers really just warm up the water. :-)
Wonder if smart people have started thinking about using data centers output water as an input into district heating networks? https://en.wikipedia.org/wiki/District_heating
As for energy use, I think Marc’s point applies in that we need to consider what power use data centers replaces. I posit that as businesses and other organizations move their computing to the cloud, a significant number of older, less power-efficient and poorly utilized servers (and routers and switches) get decommissioned. The aggregate effect of these millions of decommissioned servers seems quite significant.