I attended the Stanford Clean Slate CTO Summit last week. It was a great event organized by Guru Parulkar. Here’s the agenda:
- 12:00: State of Clean Slate — Nick McKeown, Stanford
- 12:30:00pm: Software defined data center networking — Martin Casado, Nicira
- 1:00: Role of OpenFlow in data center networking — Stephen Stuart, Google
- 2:30: Data center networks are in my way — James Hamilton, Amazon
- 3:00: Virtualization and Data Center Networking — Simon Crosby, Citrix
- 3:30:RAMCloud: Scalable Datacenter Storage Entirely in DRAM — John Ousterhout, Stanford
- 4:00: L2.5: Scalable and reliable packet delivery in data centers — Balaji Prabhakar, Stanford
- 4:45: Panel: Challenges of Future Data Center Networking–Panelists, James Hamilton, Stephen Stuart, Andrew Lambeth (VMWare), Marc Kwiatkowski (Facebook)
I presented Networks are in my Way. My basic premise is that networks are both expensive and poor power/performers. But, much more important, they are in the way of other even more important optimizations. Specifically, because most networks are heavily oversubscribed, the server workload placement problem ends up being seriously over-constrained. Server workloads need to be near storage, near app tiers, distant from redundant instances, near customer, and often on a specific subnet due to load balancer or VM migration restrictions. Getting networks out of the way so that servers can be even slightly better utilized will have considerably more impact than many direct gains achieved by optimizing networks.
Providing cheap 10Gibps to the host gets networks out of the way by enabling the hosting of many data intensive workloads such as data warehousing, analytics, commercial HPC, and MapReduce workloads. Simply providing more and cheaper bandwidth could potentially have more impact than many direct networking innovations.
Networking power/performance is unquestionably poor. I often refer to net gear as the SUV of the data center. However, the biggest gain in power efficiency that networks could enable isn’t in reducing networking power but in getting out of the way and allowing better server utilization. Networking is under 4% of the power consumption in a typical high-scale data center whereas severs are responsible for 44%. I’m arguing that the best networking power innovations are the ones that help make the use of servers more efficient.
Looking at networking cost, we see we actually do have a direct problem there. At scale, networking gear represents a full 18% of the cost of all infrastructure (shell, power, power distribution, mechanical systems, servers,& networking). For every $2.5 spend on servers, roughly $1 is spent on networking. Over time, the ratio of networking gear to servers continues to worsen. I look at this phenomena in more detail in It’s the Eco System Stupid where the commodity server ecosystem is compared to the to the current networking equipment ecosystem. In my view, the industry needs an competitive, multi-source networking hardware and software stack.
–jrh
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
From Ken Bross of Intel who was having trouble posting due to what is likely a software issue on my end:
James,
On the power/efficiency front, it’s important to have some sort of benchmark or metric that can be used to validate the efficiency of the network equipment. As network functions become more complex, it may become difficult to classify the efficiency of mixed functionality, but there are some metrics emerging for at least the simpler cases. In particular, ANSI/ATIS has released some telecom-related standards that may be of interest. Specifically:
* General Requirements: http://www.atis.org/docstore/product.aspx?id=24547
* Router and Ethernet Switch Requirements: http://www.atis.org/docstore/product.aspx?id=24667
[Full disclosure: I participated in the development of these ANSI standards.]
These standards should be applicable in the data center environment as much as in the central office environment (perhaps with the caveat that data center equipment needs the option of testing with AC power input). As data center operators seek to improve the energy efficiency of their data centers, I think metrics such as this will become more important, and leading networking equipment vendors will begin to highlight their equipment efficiency to discerning customers.
–kb