SpecPOWER is the first industry standard benchmark that evaluates power and performance characteristics of high volume servers.  Yesterday the final spec was released.

I have quibbles with this benchmark but they really are mostly quibbles.  Generally, I’m thrilled to see a benchmark out there that shines a light on server power consumption.  The entire industry needs it. Benchmarks drive action and we have lots of low hanging fruit to grab when it comes to server power/performance.

The press release is up at: http://www.spec.org/power_ssj2008/ and early results are at http://www.spec.org/power_ssj2008/results/power_ssj2008.html.

I remember working on IBM DB2 years back when we first ran TPC-A. Prior to the benchmark we had a perfectly good database management system that customers were using successfully to run some very large businesses. When we first ran TPC-A, let’s just say we didn’t like what we saw.  We got to work and ended up improving DB2 by a full factor of 10 in that release and then went on and improved by a further factor of 4 in the next release.  Yes I know the only way to improve that much is to start off REALLY needing it.  That’s my point. As a direct result of the benchmark giving us and customers increased visibility into OLTP performance, the product improved 40x in less than 5. Customers gained, the product got better, and it gave the engineering team good goals to rally around. Benchmarks help customers.

As benchmarks age, it’s harder to find the big differentiating improvements. Once the easy changes are found and even the more difficult improvements have been worked through, benchmark specials typically begin to emerge.  Companies start releasing changes that help the benchmarks and do nothing and, in rare cases, can even hurt real customer workloads. Eventually, this almost always happens and causes industry veterans, myself included, to distrust benchmarks. We forget that in the early years of most benchmarks, they really did help improve the product and delivered real value to customers.

I’m very happy to see SPEC release a benchmark that measures power and I’m confident that it will help drive big power efficiency gains. Sure, we’ll eventually see the game playing benchmark specials but, for now, I predict many of the improvements will be real. This will help the industry evolve more quickly.  Now can someone please start working on a data center efficiency benchmark?  Huge innovations are waiting in data center design and even more in how existing designs are deployed.

                                                --jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh