Perspectives - Ramblings

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Socrata Chief Technical Officer

James Hamilton — Fri, 27 Jan 2012 01:13:24 GMT

Ordinarily I focus this blog on areas of computing where I spend most of my time from high performance computing to database internals and cloud computing. An area that interests me greatly but I’ve seldom written about is entrepreneurship and startups.

One of the Seattle areas startups with which I stay in touch is Socrata. They are focused on enabling federal, state, and local governments to improve the reach, usability and social utility of their public information assets. Essentially making public information available and useful to their constituents. They are used by: the World Bank, the United Nations, the World Economic Forum, the US Data.Gov, Health & Human Services, Centers for Disease Control, several most major cities including NYC, Seattle, Chicago, San Francisco and Austin and many county and state governments. Even foreign governments like the Country of Kenya have adopted Socrata.

I first met Kevin Merritt, the founder and CEO of Socrata, back in 2005 when I was doing technical diligence for the Microsoft acquisition of the LA-based Frontbridge Technologies. I love doing diligence on startups because it’s an opportunity to dive in and spend a day or more digging deeply and understanding what smart people have produced, where things worked really well, and areas where things didn’t pan out as well as they could have. I’ve learned a lot in these roles and I’m lucky to have been able to do many of them first at IBM, later at Microsoft, and now at Amazon.

What made this one a bit different is I got a call shortly after the deal closed asking if I wanted to be the General Manager of the Microsoft subsidiary that was formed in the acquisition. An opportunity to run mid-sized business in its entirety. Development, test, operations, and customer support. Absolutely! I’ve never learned so much as I did in the first year or so at what would become Microsoft Exchange Hosted Services.

It was a great experience and I’ve been 100% focused on cloud services since that time. And, as a consequence of leading Frontbridge, I got to know Kevin Merritt well. He is an excellent strategic thinker and an even better operator. Whenever Kevin was involved, customers were happy and the service was rapidly improving and expanding. Kevin eventually left to form Socrata and he and I have stayed in touch since then. He knows I’m a sucker for a beer and some wings :-).

Based in Seattle, Socrata is venture-backed with a small and talented engineering team. They are enjoying strong customer demand and their market success is fueling growth in the engineering team. They are currently looking for a CTO and, if I didn’t already have one of the best job out there, I would seriously considering joining Kevin and the team. If you are a technology leader interested in big data, cloud computing, architecture of distributed systems, ops automation, and the user experience of making data easy to find and use, you should send Kevin, their founder and CEO, a note at kevin.merritt@socrata.com.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

AWS Startup Challenge 2011

James Hamilton — Mon, 14 Nov 2011 15:24:52 GMT

Last week I got to participate in one of my favorite days each year, serving on the judging panel for the AWS Startup Challenge. The event is a fairly intense day where our first meeting starts at 6:45am and the event closes at 9pm that evening. But it is an easy day to love in that the entire day is spent with innovative startups who have built their companies on cloud computing.

I’m a huge believer in the way cloud computing is changing the computing landscape and that’s all I’ve worked on for many years now. But I have still not tired of hearing “Without AWS, we wouldn’t have even been able to think about launching this business.”

Cloud computing is allowing significant businesses to be conceived and delivered at scale with only tiny amounts of seed funding or completely bootstrapped. Many of the finalist we looked at last week’s event had taken less than $200k of seed funding and yet had already had thousands of users. That simply wouldn’t have been possible 10 years ago and I just love to see it.

The finalist for this year’s AWS Startup Challenge were:

Booshaka - United States ( Sunnyvale, California ) >

Booshaka simplifies advocacy marketing for brands and businesses by making sense of large amounts of social data and providing an easy to use software-as-a-service solution. In an era where people are bombarded by media, advertisers face significant challenges in reaching and engaging their customers. Booshaka combines the social graph and big data technology to help advertisers turn customers into their best marketers.

Deputy.com - Australia ( Sydney ) >

Deputy.com is an online business management solution specifically addressing the HR department. The powerful online and mobile platform engages all staff across an enterprise, builds positive culture and drives business growth.

Fantasy Shopper - UK ( Exeter ) >

Fantasy Shopper is a social shopping game. The shopping platform centralizes, socializes and “gamifies” online shopping to provide a real-world experience.

Flixlab - United States ( Palo Alto, California ) >

With Flixlab, people can instantly and automatically transform raw videos and photos from their smartphone or their friends’ smartphones, into fun, compelling stories with just a few taps and immediately share them online. After creation, viewers can then interact with these movies by remixing them and creating personally relevant movies from the shared pictures and videos.

Getaround - United States ( San Francisco, California ) >

Getaround is a peer-to-peer car sharing marketplace that enables car owners to rent their cars to qualified drivers by the hour, day, or week. Getaround facilitates payment, provide 24/7 roadside assistance, and provide complete insurance backed by Berkshire Hathaway with each rental. >

Intervention Insights - United States ( Grand Rapids, Michigan ) >

Intervention Insights provides a medical information service that combines cutting edge bioinformatics tools with disease information to deliver molecular insights to oncologists describing an individual’s unique tumor at a genomic level. The company then provides a report with an evidenced-based list of therapies that target the unique molecular basis of the cancer.

Localytics - United States ( Cambridge, Massachusetts )>

Localytics is a real-time mobile application analytics service that provides app developers with tools to measure usage, engagement and custom events in their applications. All data is stored at a per-event level instead of as aggregated counts. This allows app publishers, for example, to create more accurately targeted advertising and promotional campaigns from detailed segmentation of their dedicated customers.

Judging this year’s competition was even more difficult than last year because of the high quality of the field. Rather than a clear winner just jumping out, nearly all the finalist were viable winners and each clearly led in some dimensions.

As I write this and reflect on the field of finalist, some notable aspects of the list: 1) it is truly international in that there are several very strong entrants from outside the US and more than ½ of the finalists come from outside of Silicon Valley – the combination of two trends is powerful: first the economics of cloud computing supports successful startups without venture funding and, second, the spread of venture and angel funding throughout the world. Both trends make for a very strong field. Continuing on the notable attributes list, 2) very early stage startups are getting traction incredibly quickly – cloud computing allows companies to go to beta without having to grow a huge company. And, 3) Diversity. There were consumer offerings, developer offerings, and services aimed at highly skilled professionals.

The winner of the AWS Startup Challenge this year was Fantasy Shopper from Exeter, United Kingdom. Fantasy Shopper is a small, mostly bootstrapped startup led by CEO Chris Prescott and CTO Dan Noz with two other engineers. Fantasy Shopper is a social shopping game. They just went into beta on October 18^th and already have thousands of incredibly engaged users. My favorite example of which is this video blog posted to YouTube November 6^th: http://www.youtube.com/watch?v=h_sKDgdEexk. Watch the first 60 to 90 seconds and you’ll see what I mean.

Congratulations to Chris, Dan, Brendan, and Findlay at Fantasy Shopper and keep up the great work.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Presenting Tomorrow at University of Washington

James Hamilton — Tue, 01 Nov 2011 02:07:27 GMT

I’m not sure why it all happens at once but it often does. Last Monday I kicked off HPTS 2011 in Asilomar California and then flew to New York City to present at the Open Compute Summit.

I love HPTS. It’s a once every 2 year invitational workshop that I’ve been attending since 1989. The workshop attracts a great set of presenters and attendees: HPTS 2011 agenda. I blogged a couple of the sessions if you are interested:

· Microsoft COSMOS at HPTS

· Storage Infrastructure Behind Facebook Message

The Open Compute Summit was kicked off by Frank Frankovsky of Facebook followed by the legendary Andy Bechtolsheim of Arista Networks. I did a talk after Andy which was a subset of the talk I had done earlier in the week at HPTS.

· HPTS Slides: Internet-Scale Data Center Economics: Costs and Opportunities

· OCP Summit: Internet Scale Infrastructure Innovation

Tomorrow I’ll be at University of Washington presenting Internet Scale Storage at the University of Washington Computer Science and Engineering Distinguished Lecturer Series (agenda). Its open to the public so, if you are in the Seattle area and interested, feel free to drop by EEB-105 at 3:30pm (map and directions).

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Sprint is Giving Free Customer Service Lessons

James Hamilton — Sat, 29 Oct 2011 14:40:18 GMT

Sometimes the most educational lessons are on what not to do rather than what to do. Failure and disaster can be extraordinarily educational as long as the reason behind the failure is well understood. I study large system outages, infrastructure failures, love reading post mortems (when they actually have content), and always watch carefully how companies communicate with their customers during and right after large scale customer impacting events. I don’t do it because I enjoy failure – these things all scare me. But, in each there are lessons to be learned.

Sprint advertising from: http://unlimited.sprint.com/?pid=10 (2011/10/29).

I typically point out the best example rather than the worst but every once in a while you see a blunder so big it just can’t be ignored. Sprint is the 3^rd place wireless company in an industry where backbreaking infrastructure costs strongly point towards there only being a small number of surviving companies unless services are well differentiated. All the big wireless players work hard on differentiation but it’s a challenge and, over time, the biggest revenue, supports the biggest infrastructure investment, and its gets harder and harder to be successful as a #3 player.

Sprint markets that they are better than the #1 and #2 carrier because they really have unlimited data rather than merely using the word “unlimited” in the marketing literature. They say “at Sprint you get unlimited data, no overage charges, and no slowing you down” (http://unlimited.sprint.com/?pid=10).

We live on a boat and so 4G cellular is about as close as we can get to broadband. I like to do all I can to encourage broad competition because it is good for the industry and good for customers. That is one of the reasons we are Sprint customers today. We use Sprint because they offer unlimited 4G and I really would like there to be more than 2 surviving North American wireless providers.

Unfortunately, Sprint seems less committed to my goal of keeping the #3 wireless player healthy and solvent. Looking at the Sprint primary differentiating feature, unlimited data, they plan to shut it off this month. That’s a tough decision but presumably it was made with care and there exists a plan to grow the company with some other feature or features making them a better choice than Verizon or AT&T. Just being a 3^rd choice with a less well developed network and with less capital to invest into that network doesn’t feel like a good long term strategy for Sprint.

What makes Sprint’s decision notable is the way the plan was rolled out. Sprint has many customers under 2 year, unlimited data contracts. Rather than risk the negative repercussions and customer churn from communicating the change, they went the stealth route. The only notification was buried in the fine print of the October bill:

Mobile Broadband Data Allowance Change
Eff. on your next bill, Mobile Broadband Data Plan 4G usage will be combined with your current 3G monthly data allowance and no longer be unlimited. On-network data overage rate for 3G/4G is $.05/MB. Monitor combined data usage at sprint.com. Please visit sprint.com/servicechange for details.

In November, many of us are going to get charged an overage fee of $0.05/MB on what has been advertised heavily as the only “real” unlimited plan. For many customers, the only reason they have a Sprint contract is that the data plan was uncapped. Both my phone and Jennifer’s are with AT&T. The only reason we are using Sprint for connectivity from the boat WiFi system is Sprint offered unbounded data. Attempting a stealth change of the primary advertised characteristic of a service shows very little respect for customers even when compared with other telcos, an industry not generally known for customer empathy.

I agree that almost nobody is going to read the bill and I suppose some won’t notice when subsequent bills are higher. But many eventually will. And, even for those that don’t notice and are silently are getting charged more, when they do notice, they are going to be unhappy. No matter how you cut it, the experience is going to be hard on customer trust. And, at the same time they showing little respect for customers, they are releasing them all from contract at the same time. Any Sprint customer is now welcome to leave without termination charge.

Some analysts have speculated that Sprint doesn’t have the bandwidth to support their launch of iPhone. This billing structure change strongly suggests that Sprint really does have a bandwidth problem. I’ve still not yet figured out why an iPhone is more desirable at Sprint than it is at Verizon or AT&T. And I still can’t figure out why the #3 provider with the same data caps is more desirable than the big 2 but it’s not important that I understand. That’s a Sprint leadership decision.

Let’s assume that the Sprint network is in capacity trouble and they have no choice but to cap the data plans even though they are changing the very terms they advertised as their primary advantage. Even if that is necessary, I’m 100% convince the right way to do it is to support the existing contact terms for the duration of those contracts. If the company really is teetering on failure and is unable to honor the commitments they agreed to, then they need to be upfront with customers. You can’t slip in new contract terms quietly into the statement and hope nobody notices. Showing that little respect for customers is usually rewarded by high churn rates and a continuing to shrink market share. Poor approach.

I called Sprint and pointed out they were kind of missing the original contact terms. They said “there was nothing they could do” however, they would be willing to offer a $100 credit if we would agree to another 2 year contract term. Paying only $100 to get a customer signed up for another 2 years would be an incredible bargain for Sprint. Most North American carriers spend at least that on device subsidies when getting customers committed to an additional 2 year term. This would be cheap for Sprint and would get customers back under contract after this term change effectively released them. The Sprint customer service representative did correctly offer to waive early cancellation fees since they were changing the contract terms of the original contract.

Sprint customers are now all able to walk away today from the remaining months in their wireless contacts without any cost. They are all free to leave. From my perspective, it is just plain nutty for Sprint to give their entire subscription base the freedom to walk away from contracts without charge while, at the same time, treating them poorly. It’s a recipe for industry leading churn.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Amazon Technology Open House

James Hamilton — Thu, 09 Jun 2011 13:40:21 GMT

The Amazon Technology Open House was held Tuesday night at the Amazon South Lake Union Campus. I did a short presentation on the following:

• Quickening pace of infrastructure innovation

• Where does the money go?

• Power distribution infrastructure

• Mechanical systems

• Modular & Advanced Building Designs

• Sea Change in Networking

Slides and notes:

Slides are posted at: http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_AmazonOpenHouse20110607.pdf
Coverage on Data Center Knowledge: http://www.datacenterknowledge.com/archives/2011/06/09/a-look-inside-amazons-data-centers/
Coverage on Dave Ohara on GreenM3: http://www.greenm3.com/gdcblog/2011/6/8/amazonrsquos-data-center-container-quotperdixquot-something.html
Xconomy Seattle: http://www.xconomy.com/seattle/2011/06/09/adding-a-circa-2000-amazon-com-every-day-data-centers-with-no-air-conditioning-more-from-amazon-web-services-james-hamilton/

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Atul Gawande on Performance

James Hamilton — Fri, 03 Jun 2011 15:24:06 GMT

Earlier today Alex Mallet reminded me of the excellent writing of Atul Gawnade by sending me a pointer to the New Yorker coverage of Gawande’s commencement address at the Harvard Medical School: Cowboys and Pit Crews.

Four years ago I wrote a couple of blog entries on Gawande’s work but, at the time, my blog was company internal so I’ve not posted these notes here in the past:

As a follow-on to the posting I made on professional engineering (also posted externally http://perspectives.mvdirona.com/2007/11/07/ProfessionalEngineering.aspx) Edwin Young sent me a link to the following talk by Atul Gawande: Outcomes are very Personal. It’s from another domain, medicine, but is a phenomenally good presentation by a surgeon and his core premise applies equally to software: practitioners work and the outcomes of that work are spread on a bell curve. The truly great are much better than the average and often an order of magnitude better than the lowest performing. His book and the presentation is about the personal attributes and approaches of those at the very top. It’s well worth a view: http://www.youtube.com/watch?v=MbNu6LY5sMY.

In my view, it’s an insightful presentation by a surgeon who loves data, loves understanding why we do well and how we can do better and is relentless in pursuit himself of doing everything better. Subsequent to watching the presentation, I read a book by the same author “Better: A Surgeon's Notes on Performance” (http://www.amazon.com/Better-Surgeons-Performance-Atul-Gawande/dp/0805082115).

Software, like surgery, is part art and part science and there is tremendous variability between the average and the best. Gawande studies the best in different specializations to understand why the performance of some practitioners is way out there at the positive end of the bell curve and, through a series of essays, makes observations on how to do improve performance of the population Aoverall. Understanding that human performance is distributed on the bell curve means that, for whatever it is you are doing, there are average performers, terrible performers, and truly gifted performers. Gawade looks for what he calls positive deviance – it’s always there in a bell curve distributed phenomena – and tries to understand what they do differently. Worth reading.

A sampling of Gawande’s work:

· Book: http://www.amazon.com/Better-Surgeons-Performance-Atul-Gawande/dp/0805082115

· Video: http://www.youtube.com/watch?v=MbNu6LY5sMY

· Commencement Speech Notes: http://www.newyorker.com/online/blogs/newsdesk/2011/05/atul-gawande-harvard-medical-school-commencement-address.html

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

What Went Wrong at Fukushima Dai-1

James Hamilton — Tue, 31 May 2011 13:39:33 GMT

As a boater, there are times when I know our survival is 100% dependent upon the weather conditions, the boat, and the state of its equipment. As a consequence, I think hard about human or equipment failure modes and how to mitigate them. I love reading the excellent reporting by the UK Marine Accident Investigation Board. This publication covers human and equipment related failures on commercial shipping, fishing, and recreational boats. I read it carefully and I’ve learned considerably from it.

I treat my work in much the same way. At work, human life is not typically at risk but large service failures can be very damaging and require the same care to avoid. As a consequence, at work I also think hard about possible human or equipment failure modes and how to mitigate them.

Wanting to deeply understand unusual failure modes and especially wanting to understand the errors that humans make when managing systems under stress, I spend time reading about system failures. Considerable learning can be drawn from reading about the failures of engineered systems and people under stress. All disasters or near disasters yield some unique lessons and re-enforce some old ones.

The hard part for me is getting enough detail to really learn from the situation. The press reports are often light on details partly because general audiences are not necessarily that interested but there also may be legal or competitive constraints preventing broad publication. NASA, FAA, Coast Guard and some other government reports to get to excellent detail. One analysis of system failure I learned greatly from was Feynman’s analysis of the space shuttle Challenger disaster as part of the Rogers Commission Report.

I just came across another report that is not quite a Feynman classic but it is an excellent, just-the-facts description of a large scale failure. This report, from IEEE Spectrum, titled What Went Wrong in Japan’s Nuclear Reactors outlines what happened in the eventually catastrophic disaster at Japan’s Fukushima Dai-1 nuclear facility following the Tohoku earthquake and subsequent tsunami. In this report, the terminal failures of 4 of the 6 reactors at the facility is described in more detail than other accounts of that event I’ve come across.

All disasters are unique in some dimensions. What makes Fukushima particularly unusual is these failures occurred over multiple weeks rather than the seconds to hours of many events. This one was relatively slow to develop and even slower to be brought under control. Looking forward, I suspect Fukushima will share some characteristics with Chernobyl where mitigating the environmental damage is still nowhere close to complete nearly three decades later. In 1998 the Ukraine government obtained economic aid from the European Bank for Reconstruction and Development to rebuild the failing Chernobyl sarcophagus. It is expected that yet more work will need to be done to continue to contain dangerous radioactive substances from escaping. Similarly, I expect the environmental impact of the Fukushima disaster will be fought for decades at great cost both economic and human.

In many ways Fukishima was a classic disaster where a not particularly surprising event, in this case an earthquake near Japan, started the failure and then cascading natural disaster, equipment failure, and human decisions followed to yield an outcome that every aspect of the system design sought to avoid.

I recommend reading the IEEE report linked below and my rough notes from the write-up follow:

· On March 11 an earthquake registering 9.0 magnitude was experienced off the coast of Japan

· The tsunami hit the plant destroying power distribution gear cutting off power to the Fukushima facility

· Backup generators and switch gear were also disabled by the Tsunami

· Reactor building integrity was maintained through earthquake and Tsunami and the three reactors that were active at that point where all shut down properly

· Due to the power failure and the damage to distribution gear and generators, plant cooling systems were not operating at any of the reactors nor the spent fuel rod storage pools

· Even though the nuclear reaction had been stopped in the three reactors that were operational when the tsunami hit (reactors 1, 2, & 3), considerable heat was still being created putting the reactors at risk of meltdown. Meltdown is a condition where reactor core over temperature occurs, the coolant is boiled off, the fuel rods melt and form a pool of very hot, highly radioactive fuel in the bottom of the reactor. This hot, radioactive fluid then rapidly breaks down steel and concrete in the containment vessel and possibly escapes to the environment.

· Another area of risk from the failed cooling systems are the spent nuclear fuel rod storage pools. These pools are also housed inside the reactor buildings near the primary containment vessel where the active nuclear reaction actually takes place. Although the fuel rods are no longer contributing to a nuclear reaction, they are both highly radioactive and still producing sufficient heat that active cooling is required. Without cooling these rods can heat the storage pool to the point that it boils off the cooling water and present a risk similar to the active rods inside the primary storage vessel.

· I find it surprising that both the spent rod storage and the shut down reactor cores don’t appear to fail safe and self-stabilize when cooling water is removed given the considerably higher than zero probability of power failure and the seriously negative impact of radioactive release to the environment.

· Events at Reactor #1:

o March 12, a day after the power failure, heat in the recently shutdown reactor built up until the (not circulating) cooling water began to be boiled off.

o As the water level fell, the now exposed fuel rods reacted with the steam in the primary containment vessel, and began producing hydrogen gas

o The pressure rose to dangerous levels in the primary containment vessel and operators decided to vent the primary containment vessel into the reactor building.

o The vented hydrogen gas when exposed to the relatively oxygen-rich environment in the reactor building, exploded blowing the top off the reactor building

o The explosion may have also damaged the primary containment vessel and definitely released radioactive material

o The operators chose to pump seawater into the building in an effort to control the escalating temperature inside the reactor and to avoid core meltdown

o March 29, radioactive water was found outside the reactor building

o April 5, reactor core temperatures have begun to fall indicating the system is coming back into control

o Radioactivity levels in the building are very high and operators are injecting nitrogen to reduce the likelihood of subsequent hydrogen explosions.

o May 12, TEPCO officials confirmed that the reactor had suffered a core meltdown and the bottom of the reactor building may be leaking highly radioactive water into the environment.

· Events at Reactor #3:

o March 14, 3 days after the tsunami and 2 days after the roof was blown off the Reactor #1 containment building, the same thing happened on Reactor #3

o This explosion occurred despite plant operators pumping large quantities of cooling sea water into the reactor building

o March 17, steam begins billowing from the reactor building confirming that the primary containment vessel was damaged and releasing radioactive compounds.

o Helicopters dumped water on the building and police water cannons were used to pour water down onto the building.

o Water was sprayed on the building for days with some interruptions as radiations levels rose sufficiently high that work had to be stopped.

o March 24, workers laying power cables attempting to restore power to Reactor #3 waded into highly radioactive water requiring hospitalization.

o March 28, dangerous plutonium was detected in the environment near Reactor #3.

· Events at Reactor #2:

o March 15, 4 days after the tsunami, 3 days after the roof was blown off Reactor #1, and a day after the roof was blown off Reactor #3, a serious explosion occurred at Reactor #2.

o Reactor #2 was later confirmed to have experienced at least a partial core meltdown

o March 27, highly radioactive water discovered outside of reactor building #2.

· Subsequently large quantities of uncontained radioactive water has been found throughout the multi-reactor plan and the turbine facilities are flooded as are the cabling tunnels between the buildings. Serious radioactive water leaks into the ocean have been detected and subsequently corrected in one case by injecting 6,000 liters of liquid glass into the ground near the leak.

· April 4^th, 11,500 tons of radioactive water is pumped into the ocean. This water is 100x above the legal safety limit but was pumped into the environment in the hope that the storage facilities can be used to contain waste water that is 10,000x time radioactive limit for environmental release.

· The spent fuel pools at the inactive reactors 4, 5, & 6 were all slowly overheating as a consequence of there being no cooling water. The Reactor #4 cooling pool either boiled off its water or it leaked off as a result of earthquake damage. The spent fuel rods exposed to atmosphere without cooling lead to fires inside Reactor building #4

· Outcome:

o Fukushima now rated to be as serious as the Chernobyl having been classified as a a magnitude 7 event, the worst on the International Nuclear Event Scale. However it is still consider to have released only 5 to 10% of the radiation released by Chernobyl.

o All residents within 20 km evacuated

o Voluntary evacuation of all residents between 20 and 30 km.

o Agricultural products including milk and vegetables from the region contaminated

o Tokyo’s tap water declared unfit for infants for 1 day

o Decades of cleanup and containment remain

The report: What Went Wrong in Japan's Nuclear Reactors: http://spectrum.ieee.org/tech-talk/energy/nuclear/explainer-what-went-wrong-in-japans-nuclear-reactors.

We all wish the situation had been avoided and, those of us involved in engineering projects whether they be life critical systems or not, need to ensure that the lessons from this one are learned well and applied faithfully to new designs. I won’t speculate on human risk in the efforts spent to mitigate this disaster but, clearly, the workers that brought these systems back under control and continue to manage the environmental impact are heroes and deserve our collective thanks.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

DataCloud 2010: Workshop on Data Intensive Computing in the Clouds Call for Papers

James Hamilton — Tue, 14 Sep 2010 20:45:05 GMT

For those of you writing about your work on high scale cloud computing (and for those interested in a great excuse to visit Anchorage Alaska), consider submitting a paper to the Workshop on Data Intensive Cloud Computing in the Clouds (DataCloud 2011). The call for papers is below.

--jrh

-------------------------------------------------------------------------------------------

                                           *** Call for Papers ***
      WORKSHOP ON DATA INTENSIVE COMPUTING IN THE CLOUDS (DATACLOUD 2011)
                  In conjunction with IPDPS 2011, May 16, Anchorage, Alaska
                                http://www.cct.lsu.edu/~kosar/datacloud2011
-------------------------------------------------------------------------------------------

The First International Workshop on Data Intensive Computing in the Clouds (DataCloud2011) will be held in conjunction with the 25th IEEE International Parallel and Distributed Computing Symposium (IPDPS 2011), in Anchorage, Alaska.

Applications and experiments in all areas of science are becoming increasingly complex and more demanding in terms of their computational and data requirements. Some applications generate data volumes reaching hundreds of terabytes and even petabytes. As scientific applications become more data intensive, the management of data resources and dataflow between the storage and compute resources is becoming the main bottleneck. Analyzing, visualizing, and disseminating these large data sets has become a major challenge and data intensive computing is now considered as the “fourth paradigm” in scientific discovery after theoretical, experimental, and computational science.

DataCloud2011 will provide the scientific community a dedicated forum for discussing new research, development, and deployment efforts in running data-intensive computing workloads on Cloud Computing infrastructures. The DataCloud2011 workshop will focus on the use of cloud-based technologies to meet the new data intensive scientific challenges that are not well served by the current supercomputers, grids or compute-intensive clouds. We believe the workshop will be an excellent place to help the community define the current state, determine future goals, and present architectures and services for future clouds supporting data intensive computing.

Topics of interest include, but are not limited to:

- Data-intensive cloud computing applications, characteristics, challenges
- Case studies of data intensive computing in the clouds
- Performance evaluation of data clouds, data grids, and data centers
- Energy-efficient data cloud design and management
- Data placement, scheduling, and interoperability in the clouds
- Accountability, QoS, and SLAs
- Data privacy and protection in a public cloud environment
- Distributed file systems for clouds
- Data streaming and parallelization
- New programming models for data-intensive cloud computing
- Scalability issues in clouds
- Social computing and massively social gaming
- 3D Internet and implications
- Future research challenges in data-intensive cloud computing

IMPORTANT DATES:
Abstract submission: December 1, 2010
Paper submission: December 8, 2010
Acceptance notification: January 7, 2011
Final papers due: February 1, 2011

PAPER SUBMISSION:
DataCloud2011 invites authors to submit original and unpublished technical papers. All submissions will be peer-reviewed and judged on correctness, originality, technical strength, significance, quality of presentation, and relevance to the workshop topics of interest. Submitted papers may not have appeared in or be under consideration for another workshop, conference or a journal, nor may they be under review or submitted to another forum during the DataCloud2011 review process. Submitted papers may not exceed 10 single-spaced double-column pages using 10-point size font on 8.5x11 inch pages (IEEE conference style), including figures, tables, and references. DataCloud2011 also requires submission of a one-age (~250 words) abstract one week before the paper submission deadline.

WORKSHOP and PROGRAM CHAIRS:
Tevfik Kosar, Louisiana State University
Ioan Raicu, Illinois Institute of Technology

STEERING COMMITTEE:
Ian Foster, Univ of Chicago & Argonne National Lab
Geoffrey Fox, Indiana University
James Hamilton, Amazon Web Services
Manish Parashar, Rutgers University & NSF
Dan Reed, Microsoft Research
Rich Wolski, University of California, Santa Barbara
Liang-Jie Zhang, IBM Research

PROGRAM COMMITTEE:
David Abramson, Monash University, Australia
Roger Barga, Microsoft Research
John Bent, Los Alamos National Laboratory
Umit Catalyurek, Ohio State University
Abhishek Chandra, University of Minnesota
Rong N. Chang, IBM Research
Alok Choudhary, Northwestern University
Brian Cooper, Google
Ewa Deelman, University of Southern California
Murat Demirbas, University at Buffalo
Adriana Iamnitchi, University of South Florida
Maria Indrawan, Monash University, Australia
Alexandru Iosup, Delft University of Technology, Netherlands
Peter Kacsuk, Hungarian Academy of Sciences, Hungary
Dan Katz, University of Chicago
Steven Ko, University at Buffalo
Gregor von Laszewski, Rochester Institute of Technology
Erwin Laure, CERN, Switzerland
Ignacio Llorente, Universidad Complutense de Madrid, Spain
Reagan Moore, University of North Carolina
Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory
Ian Taylor, Cardiff University, UK
Douglas Thain, University of Notre Dame
Bernard Traversat, Oracle
Yong Zhao, Univ of Electronic Science & Tech of China

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Web-Scale Database

James Hamilton — Tue, 14 Sep 2010 13:55:41 GMT

I’m dragging myself off the floor as I write this having watched this short video: MongoDB is Web Scale.

It won’t improve your datacenter PUE, your servers won’t get more efficient, and it won’t help you scale your databases but, still, you just have to check out that video.

Thanks to Andrew Certain of Amazon for sending it my way.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Gone Boating

James Hamilton — Thu, 05 Aug 2010 19:50:16 GMT

I’m taking some time off and probably won’t blog again until the first week of September. Jennifer and I are taking the boat north to Alaska. Most summers we spend a bit of time between the northern tip of Vancouver island and the Alaska border. This year is a little different for 2 reasons. First, we’re heading further north than in the past and will spend some time in Glacier Bay National Park & Preserve. The second thing that makes this trip a bit different is, weather permitting, we’ll be making the nearly thousand mile one way trip as an off shore crossing. It’ll take roughly 5 days to cover the distance running 24x7 off the cost of British Columbia.

You might ask why we would want to make the trip running 24x7 off shore when the shoreline of BC is one of the most beautiful in the world. It truly is wonderful and we do love the area. We’ve even written a book about it (Secret Coast). We’re skipping the coast and heading directly to Alaska as a way to enjoy Alaska by boat when we really can’t get enough time off work to do Alaska at a more conventional, relaxed pace. The other reason to run directly there is it’s a chance to try running 24x7 and see how it goes. Think of it as an ocean crossing with training wheels. If it gets unpleasant, we can always turn right and head to BC. And, it’ll be an adventure.

We’ll be back the first week of September. Have a good rest of your summer,

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

The Drive-by Download Problem

James Hamilton — Thu, 10 Jun 2010 13:57:45 GMT

A couple of days ago I came across an interesting article by Microsoft Fellow Mark Russinovich. In this article, Mark hunts a random Internet Explorer crash with his usual tools: The Case of the Random IE Crash. He chases down the IE issue to a Yahoo! Toolbar. This caught my interest for two reasons: 1) the debug technique used to chase it down was interesting, and 2) it’s a two week old computer with no toolbars ever installed. From Mark’s blog:

This came as a surprise because the system on which the crash occurred was my home gaming system, a computer that I’d only had for a few weeks. The only software I generally install on my gaming systems are Microsoft Office and games. I don’t use browser toolbars and if I did, would obviously use the one from Bing, not Yahoo’s. Further, the date on the DLL showed that it was almost two years old. I’m pretty diligent about looking for opt-out checkboxes on software installers, so the likely explanation was that the toolbar had come onto my system piggybacking on the installation of one of the several video-card stress testing and temperature profiling tools I used while overclocking the system. I find the practice of forcing users to opt-out annoying and not giving them a choice even more so, so was pretty annoyed at this point. A quick trip to the Control Panel and a few minutes later and my system was free from the undesired and out-of-date toolbar.

It’s a messy world out there and its very tough to control what software gets installed on a computer. This broad class of problems are generally referred to as Drive-by Downloads:

The expression drive-by download is used in four increasingly strict meanings:

1. Downloads which the user indirectly authorized but without understanding the consequences (eg. by installing an unknown ActiveX component or Java applet).

2. Any download that happens without knowledge of the user.

3. Download of spyware, a computer virus or any kind of malware that happens without knowledge of the user. Drive-by downloads may happen by visiting a website, viewing an e-mail message or by clicking on a deceptive popup window: the user clicks on the window in the mistaken belief that, for instance, an error report from the PC itself is being acknowledged, or that an innocuous advertisement popup is being dismissed; in such cases, the "supplier" may claim that the user "consented" to the download although actually unaware of having initiated an unwanted or malicious software download.

4. Download of malware through exploitation of a web browser, e-mail client or operating system vulnerability, without any user intervention whatsoever. Websites that exploit the Windows Metafile vulnerability (eliminated by a Windows update of 5 January 2006) may provide examples of "drive-by downloads" of this sort.

This morning I came across what looks like a serious case of a drive-by download where the weapon of choice was the widely trusted Windows Update: Microsoft Secretly Installs Firefox Extension Through WU.

I’m a huge fan of Windows Update – I think its dramatically improved client-side security and reliability. The combination of Windows Error Reporting and Windows Update allow system failures to be statistically tracked, focus the resources on those causing the most problems, and then deliver the fixes broadly and automatically. These two tools are incredibly important to the health Windows ecosystem so I hope this report is inaccurate.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

PUE is Still Broken and I still use it

James Hamilton — Tue, 25 May 2010 17:07:40 GMT

PUE is still broken and I still use it. For more on why PUE has definite flaws, see: PUE and Total Power Usage Efficiency. However, I still use it because it’s an easy to compute summary of data center efficiency. It can be gamed endlessly but it’s easy to compute and it does provide some value.

Improvements are underway in locking down of the most egregious abuses of PUE. Three were recently summarized in Technical Scribblings RE Harmonizing Global Metrics for Data Center Energy Efficiency. In this report from John Stanley, the following were presented:

· Total energy to include all forms of energy whether electric or otherwise (e.g. gas fired chiller must include chemical energy being employed). I like it but It’ll be a challenge to implement

· Total energy should include lighting, cooling, and all support infrastructure. We already knew this but its worth clairifying since it’s a common “fudge” employed by smaller operators

· PUE energy should be calculated using source energy. This is energy at the source prior to high voltage distribution losses and including all losses in energy production. For example, for gas plants, it’s the fuel energy used including heat losses and other inefficiencies. This one seems hard to compute with precision and I’m not sure how I could possibly figure out source energy where some power is base load power and some is from peak plants and some is from out of state purchases. This recommendation seems a bit weird.

As with my recommendations in PUE and Total Power Usage Efficiency, these proposed changes add complexity while increasing precision. Mostly I think the increased complexity is warranted although the last, computing source energy, looks hard to do and I don’t fully buy that the complexity is justified.

It’s a good short read: Technical Scribblings RE Harmonizing Global Metrics for Data Center Energy Efficiency. Thanks to Vijay Rao of AMD for sending this one my way.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

State of Public Sector Cloud Computing

James Hamilton — Tue, 25 May 2010 13:17:52 GMT

Federal and state governments are prodigious information technology users. Federal Chief Information Security Office Vivek Kundra reports that the United States government is spending $76B annually on 10,000 different systems. In a recently released report, State of Public Sector Cloud Computing, Vivek Kundra summarizes the benefits of cloud computing:

There was a time when every household, town, farm or village had its own water well. Today, shared public utilities give us access to clean water by simply turning on the tap; cloud computing works in a similar fashion. Just like the water from the tap in your kitchen, cloud computing services can be turned on or off quickly as needed. Like at the water company, there is a team of dedicated professionals making sure the service provided is safe and available on a 24/7 basis. Best of all, when the tap isn’t on, not only are you saving water, but you aren’t paying for resources you don’t currently need.

§ Economical. Cloud computing is a pay-as-you-go approach to IT, in which a low initial investment is required to get going. Additional investment is incurred as system use increases and costs can decrease if usage decreases. In this way, cash flows better match total system cost.

§ Flexible. IT departments that anticipate fluctuations in user load do not have to scramble to secure additional hardware and software. With cloud computing, they can add and subtract capacity as its network load dictates, and pay only for what they use.

§ Rapid Implementation. Without the need to go through the procurement and certification processes, and with a near-limitless selection of services, tools, and features, cloud computing helps projects get off the ground in record time.

§ Consistent Service. Network outages can send an IT department scrambling for answers. Cloud computing can offer a higher level of service and reliability, and an immediate response to emergency situations.

§ Increased Effectiveness. Cloud computing frees the user from the finer details of IT system configuration and maintenance, enabling them to spend more time on mission-critical tasks and less time on IT operations and maintenance.

§ Energy Efficient. Because resources are pooled, each user community does not need to have its own dedicated IT infrastructure. Several groups can share computing resources, leading to higher utilization rates, fewer servers, and less energy consumption.

This document defines cloud computing and describes the federal government approach and then goes on to cover 30 case studies. The case studies are the most interesting part of the report in that they provide a sampling of the public sector move to cloud computing showing its real and project are underway and substantial progress is being made.

It’s good to see the federal government showing leadership at a time when the need for federal services are undiminished but the burgeoning federal deficit needs to be brought under control. The savings possible through cloud computing are substantial and the federal IT spending base is enormous, so its particularly good to be adopting this new technology delivery platform at scale.

· Document: State of Public Sector Cloud Computing

· Executive Summary: State of Public Sector Cloud Computing

Thanks to Werner Vogels for sending this article my way.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

Solving World Problems With Economic Incentives

James Hamilton — Mon, 24 May 2010 12:40:26 GMT

Economic forces are more powerful than politics. Political change is slow. Changing laws takes time. Lobbyist water down the intended legislation. Companies find loop holes. The population as a whole, lacks the strength of conviction to make the tough decisions and stick with them.

Economic forces are far more powerful and certainly more responsive than political forces. Effectively, what I’m observing is great good can be done if there is a business model and profit encouraging it. Here’s my favorite two examples, partly because they are both doing great things and partly because they are so different in their approach, but still have the common thread of using the free market to improve the world.

Google RE<C

As a society, we can attempt to limit greenhouse gas emissions by trading carbon credits or passing laws attempting to force change but, in the end, it seems we just keep burning coal. In my view, the Google approach to tackling this problem is wonderful: invest in renewable energy technologies that can be cheaper than coal. More on the program: Google's Goal: Renewable Energy Cheaper than Coal and Plug into a Greener Grid: RE<C and RechargeIT Initiatives. They are working on solar thermal, high-altitude wind, and geo-thermal. The core idea is that, if renewable sources are cheaper than coal, economic forces would quickly make the right thing happen and we actually would stop burning coal. I love the approach but its fiendishly difficult.

Bill & Melinda Gates Foundation

Here’s a related approach. The problem set is totally different but there are some parallels with the previous example in that they are attempting to set up an economic system where it can be profitable to do good for society.

I attended a small presentation by Bill Gates about 5 years ago. By my measure, it was by far the best talk I’ve ever seen Gates gave. I suspect Bill wouldn’t agree that it was his best but it had a huge impact on me. No press was there and I saw nothing written about it afterwards, but two things caught my interest: 1) Gates’ understanding of world health problems is astoundingly deep, and 2) I loved his technique of applying free-market principles to battle world problems ranging from disease through population growth.

In this talk Bill noted that North American disease has a very profitable business model and consequently is heavily invested. Third world disease lacks a business model and, as a result, there is very little investment. It’s clear that many diseases are easy to control or even eradicate but there is no economic incentive and so there is no sustained progress. There are charity donations but no deep and sustained R&D investment since there are no obvious profits to be made. Bill proposed that we encourage business models that allows drug companies to invest R&D into third world health problems. They should be able to invest knowing they will be able make money on the outcome.

Current drug costs are driven almost exclusively by R&D costs. The manufacturing costs are quite low by comparison. Does this remind you of anything? It’s the software world all over again. So, the question that brings up is: Can we create a model where drugs are sold in huge volume at very low cost? I recall buying a copy of Unix for an IBM XT back in 1985 and it was $1,000 (Mark Williams Coherent). Today 1/10 of that will buy an O/S and many are free with the business model being built on services. Can we do the same thing to the drug world? Where else could this technique play out?

Using the free market to drive change is the most leveraged approach I’ve ever seen to drive change. Where else can we cost effectively change the economic model and drive a better outcome for society as a whole?

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

One Browser To Rule Them All

Wed, 21 Apr 2010 20:25:25 GMT

There have been times in past years when it really looked like we our industry was on track to supporting only a single relevant web browser. Clearly that’s not the case today. In a discussion with a co-working today on the importance of “other” browsers, I wanted to put some data on the table so I looked up the browser stats for this web site (http://mvdirona.com). I hadn’t looked for a while and found the distribution truly interesting:

Admittedly, those that visit this site clearly don’t represent the broader population well. Nonetheless, the numbers are super interesting. Firefox eclipsing Internet Explorer and by such a wide margin was surprising to me. You can’t see it in the data above but the IE share continues to decline. Chrome is already up to 17%.

Looking at the share data posted on Wikipedia (http://en.wikipedia.org/wiki/Usage_share_of_web_browsers#Summary_table and using the Net Market Share data) we see that IE has declined from over 91.4% to 61.4% in just 5 years. Again a surprisingly rapid change.

Focusing on client operating systems, from the skewed sample that accesses this site, we see several interesting trends: 1) Mac share continues to climb sharply at 16.6%, 2) Linux at 9%, 3) iphone, ipod and ipad in aggregate at over 5 ¼%, and 4) Android already over a ¼%.

Overall we are seeing more browser diversity, more O/S diversity, and unsurprisingly, more mobile devices.

--jrh

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

VPN over WiMAX

Sat, 17 Apr 2010 13:43:03 GMT

We live on a boat which has lots of upside but broadband connectivity isn’t one of them. As it turns out, our marina has WiFi but it is sufficiently unreliable that we needed another solution. I wish there was a Starbucks hotspot across the street – actually there is one within a block but we can’t quite pick up the signal even with an external antennae (Syrens).

WiFi would have been a nice solution but didn’t work so we decided to go with WiMAX. We have used ClearWire for over a year on the boat and, generally, it has worked acceptably well. Not nearly as fast as WiFi but better than 3G cellular. Recently ClearWire changed its name to Clear and “upgraded” the connectivity technology to full WiMAX. Unfortunately, the upgrade substantially reduced the coverage area, has been fairly unstable, and the Customer support although courteous and friendly is so far away from the engineering team that they basically just can’t make a difference no matter how hard they try.

We decided we had to find a different solution. I use AT&T 3G cellular with tethering and would have been fine with that as a solution. It’s a bit slower than Clear but its stable and coverage is very broad. Unfortunately, the “unlimited” plan we got some years ago is very limited to 5Gig/month and we move far more data than that. I can’t talk AT&T into offering a solution so, again, we needed something else.

Sprint now has a WiMAX service that offers good performance (although they can be a bit aggressive on throttling) and they have fairly broad coverage in our area and are expanding quickly (Sprint announces seven new WiMAX markets). Sprint has the additional nice feature on some modems where, if WiMAX is unavailable, it transparently falls back to 3G. The 3G service is still limited to 5Gig but, as long as we are on WiMAX a substantial portion of the month, we’re fine.

The remaining challenge was Virtual Private Networks (VPN) over WiMAX can be unstable. I really wish my work place supported Exchange RPC over HTTP (one of the coolest Outlook/Exchange features of all time). However, many companies believe that Exchange RPC over HTTP is insecure in that it doesn’t’ require 2 factor authentication. Ironically, many of these companies allow Blackberries’ and iPhones to access email without 2 factor auth. I won’t try to explain why one is unsafe and the other is fine but I think it might have something to do with the popularity of iPhones and Blackberries with execs and senior technical folks :-).

In the absence of RPC over HTTP, logging into the work network via VPN is the only answer. My work place uses Aventail but there are a million solutions out there. I’ve used many and love none. There are many reasons why these systems can be unstable, cause blue screens, and otherwise negatively impact the customer experience. But one that has been driving me especially nuts is frequent dropped connections and hangs when using the VPN over WiMAX. It appears to happen more frequently when there is more data in flight but to lose a connection every few minutes is quite common.

It turns out the problem is the default MTU on most client systems is 1500 but the WiMAX default is often smaller. It should still work and just be super inefficient but it doesn’t. For more details see http://www.amazon.com/Sierra-Wireless-Overdrive-Mobile-Hotspot/dp/B0032JTPMK.

To check Vista MTUs:

netsh interface ipv4 show subinterfaces

To change the MTU to 1400:

netsh interface ipv4 set subinterface "your vpn interface here" mtu=1400 store=persistent

I’m using an MTU of 1400 with Sprint and its working well. Thanks to Kitz.co.uk for the easy MTU update. If you are having flakey VPN support especially if running over WiMAX, check your MTU.

--jrh

James Hamilton, Amazon Web Services

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

ACM Science Cloud 2010 Call For Papers

Fri, 18 Dec 2009 19:24:52 GMT

I'm on the technical program committe for ACM Science Cloud 2010. You should consider both submitting a paper and attending the conference. The conference will be held in Chicago on June21st, 2010 colocated with ACM HPDC 2010 (High Performance Distributed Computing).

The call for papers abstracst are due Feb 22 with final papers due March 1st: http://dsl.cs.uchicago.edu/ScienceCloud2010/

Workshop Overview:

The advent of computation can be compared, in terms of the breadth and depth of its impact on research and scholarship, to the invention of writing and the development of modern mathematics. Scientific Computing has already begun to change how science is done, enabling scientific breakthroughs through new kinds of experiments that would have been impossible only a decade ago. Today's science is generating datasets that are increasing exponentially in both complexity and volume, making their analysis, archival, and sharing one of the grand challenges of the 21st century. The support for data intensive computing is critical to advancing modern science as storage systems have experienced an increasing gap between their capacity and bandwidth by more than 10-fold over the last decade. There is an emerging need for advanced techniques to manipulate, visualize and interpret large datasets. Scientific computing involves a broad range of technologies, from high-performance computing (HPC) which is heavily focused on compute-intensive applications, high-throughput computing (HTC) which focuses on using many computing resources over long periods of time to accomplish its computational tasks, many-task computing (MTC) which aims to bridge the gap between HPC and HTC by focusing on using many resources over short periods of time, to data-intensive computing which is heavily focused on data distribution and harnessing data locality by scheduling of computations close to the data.

The 1st workshop on Scientific Cloud Computing (ScienceCloud) will provide the scientific community a dedicated forum for discussing new research, development, and deployment efforts in running these kinds of scientific computing workloads on Cloud Computing infrastructures. The ScienceCloud workshop will focus on the use of cloud-based technologies to meet new compute intensive and data intensive scientific challenges that are not well served by the current supercomputers, grids or commercial clouds. What architectural changes to the current cloud frameworks (hardware, operating systems, networking and/or programming models) are needed to support science? Dynamic information derived from remote instruments and coupled simulation and sensor ensembles are both important new science pathways and tremendous challenges for current HPC/HTC/MTC technologies. How can cloud technologies enable these new scientific approaches? How are scientists using clouds? Are there scientific HPC/HTC/MTC workloads that are suitable candidates to take advantage of emerging cloud computing resources with high efficiency? What benefits exist by adopting the cloud model, over clusters, grids, or supercomputers? What factors are limiting clouds use or would make them more usable/efficient?

This workshop encourages interaction and cross-pollination between those developing applications, algorithms, software, hardware and networking, emphasizing scientific computing for such cloud platforms. We believe the workshop will be an excellent place to help the community define the current state, determine future goals, and define architectures and services for future science clouds.

b: http://blog.mvdirona.com / http://perspectives.mvdirona.com

From Perspectives.

AWS Wants You!

Tue, 15 Dec 2009 04:01:45 GMT

Want to join a startup team within Amazon Web Services? I’m deeply involved and excited about this project and another couple of talented engineers could really make a difference. We are looking for:

User Interface Software Development Engineer

We are looking for an experienced engineer with a proven track record of building high quality, AJAX enabled websites. HTML, JavaScript, AJAX, and CSS experience is critical, along with Java and Tomcat. Experience with languages such as PHP, Perl, Ruby, Python, etc. is also useful. You must have significant experience in designing highly reliable and scalable distributed systems, including building front end website facing applications. You must thrive in a hyper-growth environment where priorities shift fast, have strong OO design and implementation experience, knowledge of web protocols, and in-depth knowledge of Linux tools and Java EE architectures.

For more information: https://us-amazon.icims.com/jobs/107700/job

Senior Software Development Engineer

We are looking for a Senior Software Engineer with a strong track record of building production scalable, high end, reliable, data driven distributed website systems. You must be able to tackle tough challenges and feel strongly not only about building good software but about making that software achieve its goals in an operational reality. You must thrive in a hyper-growth environment where priorities shift fast, have strong OO design and implementation experience, knowledge of web protocols, and in-depth knowledge of Linux tools and Java EE architectures.

For more information: https://us-amazon.icims.com/jobs/109479/job

If you are interested, send a resume to aws-jobinfo-lee@amazon.com. I’m looking forward to working with you.

--jrh

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Brief Blogging Hiatus

Sat, 01 Aug 2009 23:26:27 GMT

I’ll be taking a brief hiatus from blogging during the first three weeks of August. Tomorrow we leave for China. You might wonder why we would go to China during the hottest time of the year. For example, our first stop, Xiamen, is expected to hit 95F today, which is fairly typical weather for this time of year (actually its comparable to the unusual weather we’ve been having in Seattle over the last week). The timing of the trip is driven by a boat we’re buying nearing completion in a Xiamen China boat yard: Boat Progress. The goal is to see the boat roughly 90% complete so we can catch any issues early and get them fixed before the boat leaves the yard. And, part of the adventure of building a boat, is to get a chance to visit the yard and see how they are built.

We love boating but, having software jobs, we end up working a lot. Consequently, the time we do get off, we spend boating between Olympia, Washington and Alaska. Since we seldom have the time for non-boat related travel, we figured we should take advantage of visiting China and see more than just the boat yard.

After the stop at the boat yard in Xiamen, we’ll visit Hong Kong, Guilin, Yangshou, Chengdu, and do a cruise of the Yangtze River and then travel to Xian followed by Beijing before returning home.

--jrh

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Pictures from the Fisher Plaza Data Center Fire

Fri, 10 Jul 2009 13:08:58 GMT

There have been many reports of the Fisher Plaza data center fire. An early one was the Data Center Knowledge article: Major Outage at Seattle Data Center. Data center fires aren’t as rare as any of us would like but this one is a bit unusual in that fires normally happen in the electrical equipment or switchgear whereas this one appears to have been a bus duct fire. The bus duct fire triggered the sprinkler system. Several sprinkler heads were triggered and considerable water was sprayed making it more difficult to get the facility back online quickly.

Several good pictures showing the fire damage were recently published in Tech Flash Photos: Inside the Fisher Fire.

--jrh

From: http://awards.acm.org/citation.cfm?id=1108679&srt=year&year=2008&aw=140&ao=AMTURING

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Barbara Liskov 2008 Turing Award Winner

Thu, 09 Jul 2009 16:43:43 GMT

MIT’s Barbara Liskov was awarded the 2008 Association of Computing Machinery Turing Award. The Turning award is the highest distinction in computer science and is often referred to as the Nobel price of computing. Past award winners are listed at: http://en.wikipedia.org/wiki/Turing_Award.

The full award citation:

Barbara Liskov has led important developments in computing by creating and implementing programming languages, operating systems, and innovative systems designs that have advanced the state of the art of data abstraction, modularity, fault tolerance, persistence, and distributed computing systems.

The Venus operating system was an early example of principled operating system design. The CLU programming language was one of the earliest and most complete programming languages based on modules formed from abstract data types and incorporating unique intertwining of both early and late binding mechanisms. ARGUS extended many of the CLU ideas to distributed programming, and incorporated the first versions of nested transactions to maintain predictable consistencies. Other advances include solutions elegantly combining theory and pragmatics in the areas of decentralized information flow, replicated storage and caching of persistent objects, and modular upgrading of distributed systems. Her contributions have been incorporated into the practice of programming, thereby influencing many of the most important systems used today: for programming, specification, systems design, and distributed architectures.

The cover article in the July Communications of the ACM was on the award: http://cacm.acm.org/magazines/2009/7/32083-liskovs-creative-joy/fulltext.

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Select Past Perspectives Postings

Thu, 28 May 2009 12:45:07 GMT

I’ve brought together links to select past postings and posted them to: http://mvdirona.com/jrh/AboutPerspectives/. It’s linked to the blog front page off the “about” link. I’ll add to this list over time. If there is a Perspectives article not included that you think should be, add a comment or send me email.

Talks and Presentations

Data Center Architecture and Efficiency

Service Architectures

Storage

Solid State Disks

Hard Disk Drives

Software & Services

Server Hardware

High-Scale Service Optimizations, Techniques, & Random Observations

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

1999 Mitsubishi 3000 VR4 For Sale

Tue, 19 May 2009 13:25:29 GMT

Our 1999 Mitsubishi 3000 VR4 For Sale. Black-on-black with 80,000 miles. $12,500 OBO. Fewer than 300 1999 VR-4s were produced for North America, and only 101 in black-on-black.

We love this car and hate to sell it, but are living downtown Seattle and no longer need a car. It's a beautiful machine, 320 HP, and handles incredibly well. We're often stopped on the street asking if we would sell it, and now we are.

Details and pictures at: http://www.mvdirona.com/somerset/vr4.html.

Our house in Bellevue is for sale as well: 4509 Somerset Pl SE, Bellevue, Wa. Virtual tour: http://vifp.com/presentation/video_flash.php?PresID=U6I3JN153KNBW7625R826E0KP2411X18&CustID=0&Branded=logo14909.jpg.

--jrh

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

HotPower '09 Call for Papers

Sun, 05 Apr 2009 15:18:20 GMT

The HotPower ’09 workshop will be held on October 10^th at the same venue and right before the Symposium on Operating Systems Principles (SOSP 2009) at Big Sky Resort Montana. Hotpower recognizes that power is becoming a central issue in the design of all systems from embedded systems to servers for high-scale data centers.

From http://hotpower09.stanford.edu/:

Power is increasingly becoming a central issue in designing systems, from embedded systems to data centers. We do not understand energy and its tradeoff with performance and other metrics very well. This limits our ability to further extend the performance envelope without violating physical constraints related to batteries, power, heat generation, or cooling.

HotPower hopes to provide a forum in which to present the latest research and to debate directions, challenges, and novel ideas about building energy-efficient computing systems. In addition, researchers coming to these issues from fields such as computer architecture, systems and networking, measurement and modeling, language and compiler design, and embedded systems will gain the opportunity to interact with and learn from one another.

If you are interesting in submitting a paper to HotPower: http://hotpower09.stanford.edu/cfp.html.

--jrh

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Microsoft Live Search is Kumo

Sat, 28 Mar 2009 01:18:10 GMT

There has been lots of speculation about the new name for Microsoft Search. The most prevalent speculation is that Live.com will be branded Kumo: Microsoft to Rebrand Search. Will it be Kumo?

Confirming that the Kumo brand is definitely the name that is being tested internally at Microsoft, I’ve noticed over the last week that the Search Engine Referral URL www.kumo.com has been showing up frequently as the source for searches that find this blog. I suppose the brand could be changed yet again as the Microsoft internal bits are released externally. But, having been through the hassle of a brand change and know how much testing it really does require, I suspect we’re looking at the final answer with this one.

--jrh

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Workshop on Hot Topics in Cloud Computing (HotCloud '09)

Thu, 19 Mar 2009 12:22:14 GMT

HotCloud ’09 is a workshop that will be held at the same time as USENIX ’09 (June 14 through 19, 2009). The CFP:

Join us in San Diego, CA, June 15, 2009, for the Workshop on Hot Topics in Cloud Computing. HotCloud '09 seeks to discuss challenges in the Cloud Computing paradigm including the design, implementation, and deployment of virtualized clouds. The workshop provides a forum for academics as well as practitioners in the field to share their experience, leverage each other's perspectives, and identify new and emerging "hot" trends in this area.

HotCloud '09 will be co-located with the 2009 USENIX Annual Technical Conference (USENIX '09), which will take place June 14–19, 2009. The exact date of the workshop will be set soon.

The call for paper is at: http://www.usenix.org/events/hotcloud09/cfp/.

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Navigating the Linux Kernel

Fri, 13 Mar 2009 13:36:54 GMT

Google Maps is a wonderfully useful tool for finding locations around town or around the globe. Microsoft Live Labs Seadragon is a developer tool-kit for navigating wall-sized or larger displays using pan and zoom. Here’s the same basic tiled picture display technique (different implementation) applied to navigating the Linux kernel: Linux Kernel Map.

The kernel map has a component-by-component breakdown of the entire Linux kernel for hardware interfaces up to user space system calls and most of what is in between. And it’s all navigatable using zoom and pan. I’m not sure what I would actually use the kernel map for but it’s kind of cool. If you could graphically zoom from the map to the source it might actually be a useful day-to-day tool rather than one a one-time thing.

Originally posted via Slashdot (Navigating the Linux Kernel like Google Maps) and sent my way by John Smiley of Amazon.

--jrh

· http://techhermit.wordpress.com/2008/12/11/microsofts-container-progenitor-to-leave/

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com

From Perspectives.

Amazon Web Services & Windows Live Mesh at Crunchies

Sun, 11 Jan 2009 16:52:35 GMT

Last night, TechCrunch hosted The Crunchies and two of my favorite services got awards. Ray Ozzie and David Treadwell accepted Best Technology Innovation/Achievement for Windows Live Mesh. Amazon CTO Werner Vogels accepted Best Enterprise Startup for Amazon Web Services.

Also awarded (from http://crunchies2008.techcrunch.com/)

Best Application Or Service

Get Satisfaction
Google Reader (winner)
Minted
Meebo
MySpace Music (runner-up)
Yelp

Best Technology Innovation/Achievement

Facebook Connect (runner-up)
Google Friend Connect
Google Chrome
Windows Live Mesh (winner)
Swype
Yahoo BOSS

Best Design

Animoto (runner-up)
Cooliris (winner)
Friendfeed
Infectious
Lala
Sliderocket

Best Bootstrapped Startup

BackType
GitHub (winner)
Socialcast
StatSheet
12seconds.tv (runner-up)

Most Likely To Make The World A Better Place

Akoha
Causes
CO2Stats
GoodGuide (winner)
Kiva (runner-up)
Better Place

Best Enterprise Startup
Amazon Web Services (winner)
Force.com
Google App Engine (runner-up)
Yammer
Zoho

Best International Startup

eBuddy (winner)
Fotonauts
OpenX
Vente-privee
Wuala (runner-up)

Best Clean Tech Startup

Better Place (runner-up)
Boston Power
ElectraDrive
Laurus Energy
Project Frog (winner)

Best New Gadget/Device

Android G1 (runner-up)
Ausus EEE 1000 Series
Flip MinoHD
iPhone 3G (winner)
SlingCatcher

Best Time Sink Site/Application

Mob Wars
iBowl
Tap Tap Range (winner)
Zivity
Texas Hold Em (runner-up)

Best Mobile Startup

ChaCha (runner-up)
Evernote (winner)
Posterous
Qik Skyfire
Truphone

Best Mobile Application

Google Mobile Application (runner-up)
imeem mobile (winner)
Pandora Radio
rolando
ShopSavvy
Ocarina

Best Startup Founder

Linda Avery and Anne Wojcicki (23andMe)
Michael Birch and Xochi Birch (Bebo)
Robert Kalin (Etsy)
Evan Williams, Jack Dorsey, Biz Stone (Twitter ) (winner)
Paul Buchheit, Jim Norris, Sanjeev Singh, Bret Taylor (FriendFeed ) (runner-up)

Best Startup CEO

Tony Hsieh (Zappos)
Jason Kilar (Hulu) (runner-up)
Elon Musk (SpaceX)
Andy Rubin (Android)
Mark Zuckerberg (Facebook) (winner)

Best New Startup Of 2008
Dropbox (runner-up)
FriendFeed (winner)
GoodGuide
Tapulous
Topsin Media
Yammer

Best Overall Startup In 2008

Amazon Web Services
Facebook (winner)
Android
hulu
Twitter (runner-up)

James Hamilton

Amazon Web Services

james@amazon.com

From Perspectives.

CACM Interview with Pat Selinger

Wed, 31 Dec 2008 13:35:39 GMT

In a previous posting, Pat Selinger IBM Ph.D. Fellowships, I mentioned Pat Selinger as one of the greats of the relational database world. Working with Pat was one of the reasons why leaving IBM back in the mid-90’s was a tough decision for me. In the December 2008 edition of the Communications of the ACM, an interview I did with Pat back in 2005 is published: Database Dialogue with Pat Selinger. It originally went out as an ACM Queue article.

If you haven’t checked out the CACM recently, you should. The new format is excellent and the articles are now worth reading. The magazine is regaining its old position of decades ago as a must read publication. The new CACM is excellent.

Thanks to Andrew Cencini for pointing me towards this one. I hadn’t yet read my December issues.

James Hamilton
Amazon Web Services
jrh@mvdirona.com

From Perspectives.

Bill Gates

Wed, 24 Dec 2008 22:07:17 GMT

Five or six years ago Bill Gates did a presentation to a small group at Microsoft on his philanthropic work at the Bill and Melinda Gates Foundation. It was by far and away the most compelling talk I had seen in that it was Bill applying his talent to solving world health problems with the same relentless drive, depth of understanding, constant learning, excitement and focus with which he applied himself daily (at the time) at Microsoft.

Thanks to O’Reilly, I just watched an interview with Bill by Charlie Rose which has a lot of the same character and Bill makes some of the same points as that talk I saw some years ago. Find it in Dale Dogherty post Admiring Bill Gates. It’s well worth watching.

--jrh

From Perspectives.

James Hamilton Joins Amazon.com

Sun, 14 Dec 2008 01:26:43 GMT

I’ve resigned from Microsoft and will join the Amazon Web Services team at the start of next year. As an AWS user, I’ve written thousands of lines of app code against S3, and now I’ll have an opportunity to help improve and expand the AWS suite.

In this case, I’m probably guilty of what many complain about in bloggers: posting rehashed news reported broadly elsewhere without adding anything new:

· http://www.techflash.com/microsoft/Key_data_center_architect_leaves_Microsoft_for_Amazon36057114.html

· http://www.datacenterknowledge.com/archives/2008/12/12/james-hamilton-leaving-microsoft/

· http://blogs.zdnet.com/microsoft/?p=1765

· http://www.theregister.co.uk/2008/12/12/amazon_goes_containers/

· http://blog.seattlepi.nwsource.com/amazon/archives/156853.asp

· http://www.readwriteweb.com/jobwire/2008/12/james-hamilton-leaves-micfroso.php

· http://blogs.msdn.com/stevecla01/archive/2008/12/12/dick-hardt-in-james-hamilton-out.aspx

Job changes generally bring some stress, and that’s probably why I’ve only moved between companies three times in 28 years. I worked 6 years as an auto-mechanic, 10 years at IBM, and 12 years at Microsoft. Looking back over my 12 years at Microsoft, I couldn’t have asked for more excitement, more learning, more challenges, or more trust.

I’ve had a super interesting time at Microsoft and leaving is tough, but I also remember feeling the same way when I left IBM after 10 years to join Microsoft. Change is good; change challenges; change forces humility; change teaches. I’m looking forward to it even though all new jobs are hard. Onward!

--jrh

From Perspectives.

Pat Selinger IBM Ph.D. Fellowship

Sat, 15 Nov 2008 19:10:14 GMT

Last week, IBM honored database giant Pat Selinger by creating a Ph.D. Fellowship in her name. I worked with Pat closely for many years at IBM and much of what I learned about database management systems was from Pat during those years. She was a one of the original members of the IBM System R team and is probably best known as the inventor of the cost based optimizer. Access Path Selection in a Relational Database Management Systems is a paper from that period that I particularly enjoyed.

From the IBM press release:

Pat Selinger IBM Ph.D. Fellowship: awarded to an exceptional female Ph.D. student worldwide with special focus on database design and management

Pat Selinger IBM Ph.D. Fellowship
Dr. Pat Selinger was a leading member of the IBM Research team that produced the world's first relational database system and established the basic architecture for the highly successful IBM DB2 database product family. Her innovative work on cost-based query optimization for relational databases has been adopted by nearly all relational database vendors and is now taught in virtually every university database course. In 1994, Dr. Selinger was named an IBM Fellow -- an honor accorded only to the top 50 technical experts in IBM -- and in 2004, she was inducted into the Women in Technology International Hall of Fame.

An ACM Queue interview with Pat: A conversation with Pat Selinger.

It’s great to see IBM actively supporting engineering education, particular encouraging female engineers, and recognizing Pat Salinger’s contribution to the commercial and academic database community.

--jrh

James Hamilton, Data Center Futures
Bldg 99/2428, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com

From Perspectives.

Blogging Hiatus on Perspectives Until Mid-August

Thu, 17 Jul 2008 12:53:25 GMT

Going boating: http://mvdirona.com/ so I’ll be taking a break from blogging until mid-august when I’m back and caught back up. Enjoy,

--jrh

James Hamilton, Data Center Futures
Bldg 99/2428, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com

From Perspectives.

Foo Camp 2008

Mon, 14 Jul 2008 01:58:43 GMT

I just got back from O’Reilly’s Foo Camp. Foo is an interesting conference format in that there is no set agenda. It’s basically self organized as a open space-type event but that’s not what makes it special. What makes Foo a very cool conference is the people. Lots of conferences invite good people but few invite such a diverse a set of attendees. It was a lot of fun.

Here’s a picture from Saturday night of (right to left) Jesse Robbins (Seattle entrepreneur and co-chair of O’Reilly’s Velocity conference), Pat Helland (Microsoft Developer Division Architect) and myself.

From: http://flickr.com/photos/jesserobbins/2663653582/

Foo is an invitational event loosely organized around folks O’Reilly find interesting or want to get to know better. Tim O’Reilly describes the invite process in this blog comment: http://gigaom.com/2005/08/16/foocampfighting/#comment-19709 .

The conference starts around 5pm on Friday with drinks followed by dinner. After dinner, the sign-up boards for Saturday and Sunday talk sessions are put up. When Tim O’Reilly announced the sign-up boards were up, attendees rose from their tables and casually started ambling towards the sign-up boards as though there really was no rush. We’ll just be doing some signing up over the course of the evening. We might do it now or perhaps we’ll do it later. No big rush. And then some folks break into a slight trot towards but still not really an overt run Suddenly, it’s a full raging stampede. We became a single flow than as a discrete set of individuals. The flow accelerated and then crashed up against the sign-up boards spilling to both sides with folks madly negotiating for pens, better locations, sharing sessions, a shift of 6” to one side or the other, or requesting to join two sessions together. Welcome to foo camp. It didn’t slow down much from that point for the next 44 hours.

Sessions are 1 hour long but it’s good form to share a session if you don’t need the full hour. Speakers use their judgment to sign up for a small, medium or large room. Some rooms have AV equipment and some don’t.

I shared a 1 hour ssession with Jeff Hammerbacher who leads the Facebook data team. Earlier last week Jeff announced that he will be leaving Facebok in September: http://valleywag.com/5024169/yet-another-hoodie+wearing-harvard-kid-drops-out-of-facebook. Jeff and I got a medium sized room in a tent beside the main building without A/V.

The title for my session was Where Does the Power go in Data Centers and How to get it Back? I didn’t show slides but much of what we covered is posted at: http://mvdirona.com/jrh/TalksAndPapers/JamesRH_DCPowerSavingsFooCamp08.ppt. In the session, we talked through how contemporary large data centers work first looking at power distribution. We tracked the power from the feed to the substation at 115,000 volts through numerous conversions before arriving at the CPU at 1.2 volts. We then talked about power saving server design techniques. And then the mechanical systems used to get the heat back out. In each section we discussed what could be done to improve the design and how much could be saved.

Our conclusion from the session was that power savings of nearly 4x where both possible and affordable using only current technology. For those participated in the session, thanks for your contribution and for your help. It was fun.

--jrh

James Hamilton, Data Center Futures
Bldg 99/2428, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com

From Perspectives.

Nokia to acquire Symbian and go open source

Tue, 24 Jun 2008 12:01:40 GMT

Earlier today Nokia announced it will acquire the remaining 52% share of the Symbian Limited to take over controlling interest of the mobile operating system provider with 91% of the outstanding shares. This alone is interesting but what is fascinating is they also announced their intention to open source Symbian to create “the most attractive platform for mobile innovation and drive the development of new and compelling web-enabled applications”. The press release reports the acquisition will be completed at 3.647 EUR/share at a total cost of 264m EUR. The new open source project responsible for the Symbian operating systems will be managed by the newly set up Symbian Foundation with support announced by Nokia, AT&T, Broadcom, Digia, NTT docomo, EA Mobile, Freescale, Fujitsu, LG, Motorola, Orange, Plusmo, Samsung, Sony Ericcson, ST, Symbian, Teleca, Texas Instruments, T Mobile, Vodaphone, and Wipro.

Tribute to Honor Jim Gray

Sun, 01 Jun 2008 18:08:01 GMT

Yesterday the Tribute to Honor Jim Gray was held at the University of California at Berkeley. We all miss Jim deeply so it really is a tough topic. But it was great to get together with literally 100s of Jim’s friends and share stories and talk about some of his accomplishments, his contributions to the field, and his contributions to each of us. Jim is amazing across all three dimensions but what is most remarkable is the profound way he helped others achieve more throughout the industry. We’re all better engineers, researchers, and human beings for having been lucky enough to have known and worked with Jim.

Also announced yesterday was the creation of the Jim Gray Chair at Cal Berkeley. Bill Gates Eric Schmidt, Marc Benioff, and Mike Stonebraker each donated $250,000 which were matched by a $1,000,000 from the Hewlett Foundation.

Seattle PI coverage: Gathering in Berkeley, Calif., today to honor legendary scientist, Microsoft researcher Jim Gray.

The morning, general session agenda:

• Welcome - Shankar Sastry

• Opening Remarks - Joseph Hellerstein

• A Tribute, Not a Memorial: Understanding Ambiguous Loss - Pauline Boss

• The Amateur Search - Michael Olson

• Jim Gray at Berkeley - Michael Harrison

• Knowledge and Wisdom - Pat Helland

• Why Did Jim Gray Win the Turing Award? - Michael Stonebraker

• Jim Gray Chair - Stuart Russell

• 500 Special Relationships: Jim as a Mentor to Faculty and Students - Ed Lazowska

• Jim Gray: His Contributions to Industry - David Vaskevitch

• A "Gap Bridger" - Richard Rashid

• Thanks to the U.S. Coast Guard - Paula Hawthorn

The afternoon, technical session agenda:

· Welcome - Shankar Sastry

· Opening Remarks - Joseph Hellerstein

· A Tribute, Not a Memorial: Understanding Ambiguous Loss - Pauline Boss

· The Amateur Search - Michael Olson

· Jim Gray at Berkeley - Michael Harrison

· Knowledge and Wisdom - Pat Helland

· Why Did Jim Gray Win the Turing Award? - Michael Stonebraker

· Jim Gray Chair - Stuart Russell

· 500 Special Relationships: Jim as a Mentor to Faculty and Students - Ed Lazowska

· Jim Gray: His Contributions to Industry - David Vaskevitch

· A "Gap Bridger" - Richard Rashid

· Thanks to the U.S. Coast Guard - Paula Hawthorn

The event was video recorded and streamed via http://webcast.berkeley.edu/ .

Update: the video will be at: Tribute to Honor Jim Gray - General Session (thanks to George Spix for sending my way).

Second Update: A good article by John Markoff of the NY Times: A Tribute to Jim Gray: Sometimes Nice Guys Do Finish First.

>> >>

--jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com

From Perspectives.

Mike Burrows: Early Days at DEC, through Microsoft, to Google

Mon, 05 May 2008 12:32:43 GMT

I forget what brought it up but sometime back Sriram Krishnan forwarded me this article on Mike Burrows and his work through Dec, Microsoft, and Google (The Genius: Mike Burrows' self-effacing journey through Silicon Valley). I enjoyed the read. Mike has done a lot over the years but perhaps his best known works of recent years are Alta Vista at DEC and Chubby at Google.

I first met Mike when he was at Microsoft Research. He and Ted Wobber (also from Digital) came up to Redmond to visit. Back then I led the SQL Server relational engine development team which included the full text search index support. I was convinced then, and still am today, that relational database engines do a good job of managing structured data but a poor job of the other 90 to 95% of the data in the world that is less structured. It just seems nuts to me that customers industry-wide are spending well over $10B a year on relational database management systems and yet only being able to effectively use these systems to manage a tiny fraction of their data. As an increasing fraction of the structured data in the world is already stored in relational database managements systems, industry growth will come from helping customers manage their less structured data.

To be fair, most RDMBS (including SQL Server) do support full text indexing but what I’m after is deep support for full text where the index is a standard access method rather than a separate indexing engine on the side and, more importantly, full statistics are tracked on the full text corpus allowing the optimizer to make high quality decisions on join orders and techniques that include full text indices.

If you haven’t read Mike’s original Chubby paper, do that: http://labs.google.com/papers/chubby.html. Another paper is at: http://labs.google.com/papers/paxos_made_live.html. Chubby is an interesting combination of name server, lease manager, and mini-distributed file system. It’s not the combination of functionality that I would have thought to bring together in a single system but it’s heavily used and well regarded at Google. Unquestionably a success.

--jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com

From Perspectives.

Blog Data Corruption

Sat, 12 Apr 2008 19:16:29 GMT

The only thing worse than no backups is restoring bad backups. A database guy should get these things right. But, I didn’t, and earlier today I made some major site-wide changes and, as a side effect, this blog was restored to December 4^th, 2007. I’m working on recovering the content and will come up with something over the next 24 hours. However it’s very likely that comments between Dec 4^th and earlier today will be lost. My apologies.

Update 2008.04.13: I was able to restore all content other than comments between 12/4/2007 and yesterday morning. All else is fine. I'm sorry about the RSS noise during the restore and for the lost comments. The backup/restore procedure problem is resolved. Please report any broken links or lingering issues. Thanks,

-jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh | blog:http://perspectives.mvdirona.com

From Perspectives.

Golden Shield Project

Wed, 09 Apr 2008 07:13:57 GMT

What’s commonly referred to as the Great Firewall of China isn’t really a firewall at all. I recently came across an Atlantic Monthly article investigating how the Great Firewall works and what it does (see The Connection has been Reset).

The official name of what is often called the Great Firewall of China is the Golden Shield project. Rather than acting as a firewall, it’s actually mirroring content and manipulating DNS, connection management, and URL redirection to implement its goal of restricting what internet users inside China can access.

This project has been widely criticized on political and social fronts – I won’t repeat them here. It’s also been widely criticized on technical grounds as ineffective, weak, and easy to thwart. Again, not my focus. This article simply caught my interest technically as content filtering at this scale is an incredibly difficult task. What techniques are employed?

Like many software security problems, no single solution solves the problem fully and the main goal of the Golden Shield project is to add friction. If it’s painful enough to get to the content they are trying to prevent from being accessed, few will bother to access it. Essentially the goal of the four levels of protection they are using is to add friction and it’s friction rather than prevention that ensures that few Chinese internet users see restricted content in any quantity. The four levels of protection/restriction are:

1. DNS Block: sites that are on the current blacklist get DNS resolution failure or get redirected to other content. This was the technique employed against google.cn to force them add filtering to their web index. For some time , all access to google.cn was redirected to their larger Chinese competitor baidu. The other application of this technique is to return DNS lookup failure so, for example, searches for http://www.illegalsite.com will return “not found”.

2. Connect: In parallel with connection requests leaving China, they are inspected. If the IP address is on the current IP blacklist, connection reset will be sent which will cause the connection to fail.

3. URL Block: If the URL contains words on the illegal word blacklist, the connection is redirected infinitely. I’m not sure if they are only sniffing the URL or also doing reverse DNS to get the site name as well but, if unacceptable words are found in the URL, they redirect the connection repeated. Some browsers hang while others return an error message.

4. Content Block: At this level the DNS lookup has been successful and the connection has been made and content is being returned to the user. As the content is returned to the requesting user inside China, it’s being scanned in parallel for unapproved keywords and phrases. If any are found, the connection is broken immediately. As well as breaking the connection mid-way, subsequent requests from that client IP to that destination IP are blocked. The first block is short, but consecutive attempts drive up the length of the IP-to-IP connect block period and may eventually draw official scrutiny.

In addition to these techniques to block access to content outside-of-China, an estimate 30,000 censors scan and get removed unapproved content posted within within China (see http://en.wikipedia.org/wiki/Internet_censorship_in_the_People%27s_Republic_of_China).

The Golden Shield project is reportedly also being used in the opposite direction to prevent access to some content inside of China from outside the country.

There are many means of subverting the Golden Shield including using a proxy server outside of China or setting up a VPN connection to a server outside of the country. Encrypted connections will also get through as well encrypted email. However, all these techniques are non-default and require some work on behalf of the user. Most users don’t bother so, for the most part, the goals of the Golden Shield are attained even though it’s technically not that strong.

The Atlantic Monthly article: http://www.theatlantic.com/doc/200803/chinese-firewall?reddit

Wired Article: http://www.wired.com/politics/security/magazine/15-11/ff_chinafirewall

Wikipedia article: http://en.wikipedia.org/wiki/Internet_censorship_in_the_People%27s_Republic_of_China

--jrh

Thanks to Jennifer Hamilton and Mitch Wyle for pointing out the Atlantic Monthly article.

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

From Perspectives.

The Box: How the Shipping Container Made the World Smaller

Mon, 24 Mar 2008 18:42:00 GMT

I’m long been a big fan of modular data centers using ISO standard Shipping containers as the component building block:

Commodity Datacenter Design: http://research.microsoft.com/~jamesrh/TalksAndPapers/JamesRH_CIDR.doc.
Commodity Datacenter Design Slides: http://research.microsoft.com/~jamesrh/TalksAndPapers/JamesRH_Amazon.ppt.

Containers have revolutionized shipping and are by far the cheapest way to move good over sea, land, rail or truck. I’ve seen them used to house telecommunications equipment, power generators, and even stores and apartments have been made using them: http://www.treehugger.com/files/2005/01/shipping_contai.php.

The datacenter-in-a-box approach to datacenter design is beginning to be deployed more widely with Lawrence Berkeley National Lab having taken delivery of a Sun Black Box and a “customer in eastern Washington” having taken delivery of a Rackable Ice Cube Module earlier this year.

Last summer I came across a book on Shipping Containers by Marc Levinson: The Box: How the Shipping Container Made the World Smaller and the World Economy Bigger. It’s a history of containers from the early experiments in 1956 through to mega-containers terminals distributed throughout the world. The book doesn’t talk about all the innovative applications of containers outside of shipping but does give an interesting background on their invention, evolution, and standardization.

-jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

From Perspectives.

Stay Hungry. Stay Foolish

Wed, 19 Mar 2008 07:55:59 GMT

Earlier today I viewed Steve Jobs 2005 Commencement Speech at Stanford University. In this talk Jobs recounts three stories and ties them together with a common theme. The first was dropping out of Reed College and showing up for the courses he wanted to take rather than spend time on those he had to take. Dropping out was a tough decision but some of what he learned in these audited courses had a fundamental impact on the Mac and, in retrospect, appeared to be a good decision or at least one that lead to a good outcome. Getting fired from Apple was the second. Clearly not his choice, not what he would have wanted to happen but it lead to Pixar, Next and rejoining Apple stronger and more experienced than before. Again, a tough path but one that may have lead to a better overall outcome. Likely he is a better and more capable leader for the experience. Finally, facing death. Death awaits us all and, when facing death, it becomes clear what really matters. It becomes clear that following your heart is what is really important. Don’t be trapped by Dogma, don’t live other people’s lives, and have the courage to follow your own intuition. Clearly nobody wants to approach to death but knowing it is coming can free each of us to realize we can’t hide, we don’t have forever, and those things that scare us most are really tiny and insignificant when compared with death. Facing death can free us to take chances and to do what is truly important even if success looks uncertain or the risk is high.

The theme that wove these three stories together and Jobs parting words for his listeners was to “stay hungry and stay foolish”.

It’s a good read: http://news-service.stanford.edu/news/2005/june15/jobs-061505.html. Or you can view it at: http://www.youtube.com/watch?v=D1R-jKKp3NA.

Sent my way by Michael Starbird-Valentine.

--jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

From Perspectives.

Kinetic Art

Sun, 16 Mar 2008 07:56:42 GMT

Theo Jansen is a Dutch artist and engineer. His work is truly amazing. What he builds are massive synthetic animals, many more than a floor high, that walk. Their gait is surprisingly realistic and they are wind powered. More than anything they are spooky and yet deeply engaging.

Check out a selection of Jansen’s work on Marni’s blog: http://delvecem.spaces.live.com/blog/cns!58EB8CB464718427!148.entry.

--jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

From Perspectives.

Email & Time Management

Fri, 22 Feb 2008 08:09:06 GMT

I get several hundred emails a day, some absolutely vital and needing prompt action and some about the closest thing to corporate spam. I know I’m not alone. I’ve developed my own systems on managing the traffic load and, on different days, have varying degrees of success in sticking to my systems. In my view, it’s important not to confuse “processing email” with what we actually get paid to do. Email is often the delivery vehicle for work needing to be done and work that has been done, but email isn’t what we “do”.

We all need to find ways of coping with all the email while still getting real work done and having a shot at a life outside of work. My approach is fairly simple:

· Don’t process email in real time or it’ll become your job. When I’m super busy, I process email twice a day: early in the morning and again in the evening. When I’m less heavily booked, I’ll try to process email in micro bursts rather than in real time. It’s more efficient and allows more time to focus on other things.

· Shut off email arrival sounds and the “new mail” toast or you’ll end up with 100 interruptions an hour and get nothing done but email.

· I get up early and try to get my email down to under 10 each morning. I typically fail but get close. And I hold firm on that number once a week. Each weekend I do get down to less than 10 messages. If I enter the weekend with hundreds of email items, I get to work all weekend. This is a great motivator to not take a huge number of unprocessed email messages into the weekend.

· Do everything you can to process a message fully in one touch. I work hard to process email once. As I work through it, I delete or respond to everything I can quickly. Those that really do require more work I divide into two groups: 1) those I will do today or, at the very latest by end of week, I flag with a priority and leave in my inbox for processing later in the day (many argue these should be moved to a separate folder and they may be right). The longer-lived items go into my todo list and I remove them from my inbox. Because I get my email down to under 10 each week and spend as much of my weekend as needed to do this, I’m VERY motivated to not have many emails hanging around waiting to be processed. Consequently most email is handled up front as I see them and the big things are moved to the todo list. Very few are prioritized for handling later in the day.

· I chose not to use rules to auto-file email. Primarily I found that if I sent email directly to another folder, I almost never looked at it. So I let everything come into my inbox and I deal with them very quickly and, for the vast majority, they will only be touched once. If I really don’t even want to see them once, I just don’t subscribe or ask not to get them.

· Set your draft folder to be your inbox. With email systems that use a separate folder for unsent mail, there is risk that you’ll get a message 90% written and ready to be sent, get interrupted and then forget to send it. I set my draft folder to be my inbox so I don’t lose unsent email. Since my email is worked down to under 10 daily, I’ll find it there for sure before end of day.

· Don’t bother with complicated folder hierarchies—they are time-consuming to manage. If you want to save something, save it in a single folder or simple folder hierarchy and let desktop search find it when you need it. Don’t waste time filing in complex ways.

· Finally, be realistic: if you can’t process at the incoming rate, it’ll just keep backing up indefinitely. If you aren’t REALLY going to read it, then delete it or file it on the first touch. Filing it has some value in that, should you start to care more in the future, you can find it via full text search and read it then.

Jeff Johnson of MSN pointed out this excellent talk on email management called “Inbox Zero” by Merlin Mann: http://www.43folders.com/2007/07/25/merlins-inbox-zero-talk. Merlin’s advice is good and he presents well.

--jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com