Kevin Clark, Director of IT Operations at Lucasfilm was interviewed by On-Demand Enterprise in We’ve Come a Long Way Since Star Wars. His organization owns IT for LucasArts, Lucasfilm, and Industrial Light and Magic.
Lucasfilm runs a 4,500 server dedicated rendering farm and they expand this farm with workstations when they are not in use to 5,500 servers in total. The servers are dual socket, dual core Opterons with 32GB of memory. Nothing unusual except the memory configuration is a bit larger than the current average. They have 400TB of storage and produce 10 to 20TB of new and changed data each day.
Clark expects the big investment next year is making their datacenter more efficient. Partly for environmental reasons and partly because, like all businesses, they are power and cooling rather than floor space constrained. This is becoming the number one issue industry-wide and I’m glad to see. Current data center designs leave a lot of room for improvement. At this year’s Foo Camp, I lead a short session on large scale data center power consumption: Where Does the Power Go and What Can We Do About It?
This cluster is medium sized but the data change rate is unusually high at 10 to 20TB a day. It’s mostly batch work with each job being quite large. It would be interesting to see more detail on the workload scheduler they have written to manage this workload. It’s a bit ironic that IBM MVS (now called Z/OS) had a great scheduler 40 years ago. In the 10 years I worked for IBM, they constantly were requesting that a high quality batch scheduler be added to AIX. And in the 11 years I’ve worked at Microsoft, there has been great interest in improving batch scheduling to the MVS-like levels. More recently, Apache Hadoop has been used to run mega-jobs and, guess what? It too needs a high-quality, prioritized, multi-job scheduler. At the Hadoop Summit, Yahoo said they are working on one. They typically contribute their Hadoop work to open source so Hadoop may have a better scheduler coming.
Thanks to Jeff Hammerbacher for pointing me to the note on Lucasfilm.
–jrh
James Hamilton, Data Center Futures
Bldg 99/2428, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com