I did a talk at Velocity 2010 last week. The slides are posted at Datacenter Infrastructure Innovation and the video is available at Velocity 2010 Keynote. Urs Holze Google Senior VP of infrastructure also did a Velocity keynote. It was an excellent talk and is posted at Urs Holzle at Velocity 2010. Jonathan Heilliger, Facebook VP of Technical Operations spoke at Velocity as well. A talk summary is up at: Managing Epic Growth in Real Time. Tim O’Reilly did a talk: O’Reilly Radar. Velocity really is a great conference.
Last week I posted two quick notes on Facebook: Facebook Software Use and 60,000 Servers at Facebook. Continuing on that theme, a few other Facebook Data points that I have been collecting of late:
From Qcon 2010 in Beijing (April 2010): memcached@facebook:
· How big is Facebook:
o 400m active users
o 60m status updates per day
o 3b photo uploads per month
o 5b pieces of content shared each week
o 50b friend graph edges
§ 130 friend per user on average
o Each user clicks on 9 pieces of content each month
· Thousands of servers in two regions [jrh: 60,000]
· Memcached scale:
o 400m gets/second
o 28m sets/second
o 2T cached items
o Over 200 TB
o Networking scale:
§ Peak rx: 530m pkts/second (60GB/s)
§ Peak tx: 500m pkts/second (120GB/s)
· Each memcached server:
o Rx: 90k pkts/sec (9.7MB/s)
o Tx 94k pkts/sec (19 MB/s)
o 80k gets/second
o 2k sets/s
o 200m items
· Phatty Phatty Multiget
o Php is single threaded and synchronous so need to get multiple objects in a single request to be efficient and fast
· Cache segregration:
o Different objects have different lifetimes so separate out
· Incast problem:
o The use of multiget increased performance but lead to incast problem
The talk is full of good data and worth a read.
From Hadoopblog, Facebook has the world’s Largest Hadoop Cluster:
- 21 PB of storage in a single HDFS cluster
- 2000 machines
- 12 TB per machine (a few machines have 24 TB each)
- 1200 machines with 8 cores each + 800 machines with 16 cores each
- 32 GB of RAM per machine
- 15 map-reduce tasks per machine
The Yahoo Hadoop cluster is reported to be twice the node count of the Facebook cluster at 4,000 nodes: Scaling Hadoop to 4000 nodes at Yahoo!. But, it does have less disk:
· 4000 nodes
· 2 quad core Xeons @ 2.5ghz per node
· 4x1TB SATA disks per node
· 8G RAM per node
· 1 gigabit ethernet on each node
· 40 nodes per rack
· 4 gigabit ethernet uplinks from each rack to the core
· Red Hat Enterprise Linux AS release 4 (Nahant Update 5)
· Sun Java JDK 1.6.0_05-b13
· Over 30,000 cores with nearly 16PB of raw disk!
http://www.bluedavy.com/iarch/facebook/facebook_architecture.pdf
http://www.qconbeijing.com/download/marc-facebook.pdf
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
Thanks Thomas and Simone. Thomas, you were asking about the cost of money. In a weak economy I use 5% but a good case can be made for other numbers. My recommendation is to use whatever your company’s finance group uses. The spreadsheet is posted at: //perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx.
I need to post the updated spreadsheet — there have been quite a few changes over the last 18 to 24 months.
–jrh
jrh@mvdirona.com
Enjoyed the video, great information. I was interested in how you determined to use the 5% cost of money for your decision making? This is a common challenges for anyone doing cost analysis. Do you ever use anything other than 5%?
Great post, and great videos. Thanks for sharing :)