Perspectives - SPECpower Released

Wednesday, December 12, 2007

SpecPOWER is the first industry standard benchmark that evaluates power and performance characteristics of high volume servers. Yesterday the final spec was released (see below thanks to Kushagra Vaid of GFS H/W Architecture & Standards).

I have quibbles with this benchmark but they really are mostly quibbles. Generally, I’m thrilled to see a benchmark out there that shines a light on server power consumption. The entire industry needs it. Benchmarks drive action and we have lots of low hanging fruit to grab when it comes to server power/performance.

The press release is up at: http://www.spec.org/power_ssj2008/ and early results are at http://www.spec.org/power_ssj2008/results/power_ssj2008.html.

I remember working on IBM DB2 years back when we first ran TPC-A. Prior to the benchmark we had a perfectly good database management system that customers were using successfully to run some very large businesses. When we first ran TPC-A, let’s just say we didn’t like what we saw. We got to work and ended up improving DB2 by a full factor of 10 in that release and then went on and improved by a further factor of 4 in the next release. Yes I know the only way to improve that much is to start off REALLY needing it. That’s my point. As a direct result of the benchmark giving us and customers increased visibility into OLTP performance, the product improved 40x in less than 5. Customers gained, the product got better, and it gave the engineering team good goals to rally around. Benchmarks help customers.

As benchmarks age, it’s harder to find the big differentiating improvements. Once the easy changes are found and even the more difficult improvements have been worked through, benchmark specials typically begin to emerge. Companies start releasing changes that help the benchmarks and do nothing and, in rare cases, can even hurt real customer workloads. Eventually, this almost always happens and causes industry veterans, myself included, to distrust benchmarks. We forget that in the early years of most benchmarks, they really did help improve the product and delivered real value to customers.

I’m very happy to see SPEC release a benchmark that measures power and I’m confident that it will help drive big power efficiency gains. Sure, we’ll eventually see the game playing benchmark specials but, for now, I predict many of the improvements will be real. This will help the industry evolve more quickly. Now can someone please start working on a data center efficiency benchmark? Huge innovations are waiting in data center design and even more in how existing designs are deployed.

--jrh

From: osgmembers-request@spec.org [mailto:osgmembers-request@spec.org] On Behalf Of Alan Adamson
Sent: Tuesday, December 11, 2007 8:44 AM
To: osgmembers@spec.org
Subject: (osgmembers-363) SPECpower benchmark availability

I am delighted to announce the public availability of the initial SPECpower benchmark, SPECpower_ssj2008 - you can find the initial press release here http://www.spec.org/power_ssj2008/ , and see initial results here :
http://www.spec.org/power_ssj2008/results/power_ssj2008.html

This benchmark release is the result of a long and strenuous process (I cannot recall a committee meeting four days a week for several hours each day), needed partly because of the new ground being plowed.

I expect that all forthcoming benchmark development groups will be thinking about how to incorporate power measurements, and that the power committee will be looking at extending the scope of its workloads, and also helping other committees learn from their experience.

And now, on to several other benchmarks in or heading for general membership review. This is a pretty exciting time for the OSG!

My thanks and congratulations to those who worked so hard on this initial SPECpower benchmark.

Alan Adamson 905-413-5933 Tieline 969-5933 FAX: 905-413-4854
OSG Chair, SPEC Internet: adamson@ca.ibm.com
Java Technology Centre
IBM Toronto Lab