I attended the Stanford Clean Slate CTO Summit last week. It was a great event organized by Guru Parulkar. Here’s the agenda:

• 12:00: State of Clean Slate -- Nick McKeown, Stanford
• 12:30:00pm: Software defined data center networking -- Martin Casado, Nicira
• 1:00: Role of OpenFlow in data center networking -- Stephen Stuart, Google
• 2:30: Data center networks are in my way -- James Hamilton, Amazon
• 3:00: Virtualization and Data Center Networking -- Simon Crosby, Citrix
• 3:30:RAMCloud: Scalable Datacenter Storage Entirely in DRAM  -- John Ousterhout, Stanford
• 4:00: L2.5:  Scalable and reliable packet delivery in data centers -- Balaji Prabhakar, Stanford
• 4:45: Panel: Challenges of Future Data Center Networking--Panelists, James Hamilton, Stephen Stuart, Andrew Lambeth (VMWare), Marc Kwiatkowski (Facebook)

I presented Networks are in my Way. My basic premise is that networks are both expensive and poor power/performers. But, much more important, they are in the way of other even more important optimizations. Specifically, because most networks are heavily oversubscribed, the server workload placement problem ends up being seriously over-constrained.  Server workloads need to be near storage, near app tiers, distant from redundant instances, near customer, and often on a specific subnet  due to load balancer or VM migration restrictions. Getting networks out of the way so that servers can be even slightly better utilized will have considerably more impact than many direct gains achieved by optimizing networks.

Providing cheap 10Gibps to the host gets networks out of the way by enabling the hosting of many data intensive workloads such as data warehousing, analytics, commercial HPC, and MapReduce workloads. Simply providing more and cheaper bandwidth could potentially have more impact than many direct networking innovations.

Networking power/performance is unquestionably poor. I often refer to net gear as the SUV of the data center. However, the biggest gain in power efficiency that networks could enable isn’t in reducing networking power but in getting out of the way and allowing better server utilization. Networking is under 4% of the power consumption in a typical high-scale data center whereas severs are responsible for 44%. I’m arguing that the best networking power innovations are the ones that help make the use of servers more efficient.

Looking at networking cost, we see we actually do have a direct problem there. At scale, networking gear represents a full 18% of the cost of all infrastructure (shell, power, power distribution, mechanical systems, servers,&  networking). For every $2.5 spend on servers, roughly $1 is spent on networking. Over time, the ratio of networking gear to servers continues to worsen. I look at this phenomena in more detail in It's the  Eco System Stupid where the commodity server ecosystem is compared to the to the current networking equipment ecosystem.  In my view, the industry needs an competitive, multi-source networking hardware and software stack.

                                                --jrh

James Hamilton

e: jrh@mvdriona.com

w: http://www.mvdirona.com

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com