I’m taking some time off and probably won’t blog again until the first week of September. Jennifer and I are taking the boat north to Alaska. Most summers we spend a bit of time between the northern tip of Vancouver island and the Alaska border. This year is a little different for 2 reasons. First, we’re heading further north than in the past and will spend some time in Glacier Bay National Park & Preserve. The second thing that makes this trip a bit different is, weather permitting, we’ll be making the nearly thousand mile one way trip as an off shore crossing. It’ll take roughly 5 days to cover the distance running 24x7 off the cost of British Columbia.
You might ask why we would want to make the trip running 24x7 off shore when the shoreline of BC is one of the most beautiful in the world. It truly is wonderful and we do love the area. We’ve even written a book about it (Secret Coast). We’re skipping the coast and heading directly to Alaska as a way to enjoy Alaska by boat when we really can’t get enough time off work to do Alaska at a more conventional, relaxed pace. The other reason to run directly there is it’s a chance to try running 24x7 and see how it goes. Think of it as an ocean crossing with training wheels. If it gets unpleasant, we can always turn right and head to BC. And, it’ll be an adventure.
We’ll be back the first week of September. Have a good rest of your summer,
--jrh
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
A couple of days ago I came across an interesting article by Microsoft Fellow Mark Russinovich. In this article, Mark hunts a random Internet Explorer crash with his usual tools: The Case of the Random IE Crash. He chases down the IE issue to a Yahoo! Toolbar. This caught my interest for two reasons: 1) the debug technique used to chase it down was interesting, and 2) it’s a two week old computer with no toolbars ever installed. From Mark’s blog:
This came as a surprise because the system on which the crash occurred was my home gaming system, a computer that I’d only had for a few weeks. The only software I generally install on my gaming systems are Microsoft Office and games. I don’t use browser toolbars and if I did, would obviously use the one from Bing, not Yahoo’s. Further, the date on the DLL showed that it was almost two years old. I’m pretty diligent about looking for opt-out checkboxes on software installers, so the likely explanation was that the toolbar had come onto my system piggybacking on the installation of one of the several video-card stress testing and temperature profiling tools I used while overclocking the system. I find the practice of forcing users to opt-out annoying and not giving them a choice even more so, so was pretty annoyed at this point. A quick trip to the Control Panel and a few minutes later and my system was free from the undesired and out-of-date toolbar.
It’s a messy world out there and its very tough to control what software gets installed on a computer. This broad class of problems are generally referred to as Drive-by Downloads:
The expression drive-by download is used in four increasingly strict meanings:
1. Downloads which the user indirectly authorized but without understanding the consequences (eg. by installing an unknown ActiveX component or Java applet).
2. Any download that happens without knowledge of the user.
3. Download of spyware, a computer virus or any kind of malware that happens without knowledge of the user. Drive-by downloads may happen by visiting a website, viewing an e-mail message or by clicking on a deceptive popup window: the user clicks on the window in the mistaken belief that, for instance, an error report from the PC itself is being acknowledged, or that an innocuous advertisement popup is being dismissed; in such cases, the "supplier" may claim that the user "consented" to the download although actually unaware of having initiated an unwanted or malicious software download.
4. Download of malware through exploitation of a web browser, e-mail client or operating system vulnerability, without any user intervention whatsoever. Websites that exploit the Windows Metafile vulnerability (eliminated by a Windows update of 5 January 2006) may provide examples of "drive-by downloads" of this sort.
This morning I came across what looks like a serious case of a drive-by download where the weapon of choice was the widely trusted Windows Update: Microsoft Secretly Installs Firefox Extension Through WU.
I’m a huge fan of Windows Update – I think its dramatically improved client-side security and reliability. The combination of Windows Error Reporting and Windows Update allow system failures to be statistically tracked, focus the resources on those causing the most problems, and then deliver the fixes broadly and automatically. These two tools are incredibly important to the health Windows ecosystem so I hope this report is inaccurate.
--jrh
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
PUE is still broken and I still use it. For more on why PUE has definite flaws, see: PUE and Total Power Usage Efficiency. However, I still use it because it’s an easy to compute summary of data center efficiency. It can be gamed endlessly but it’s easy to compute and it does provide some value.
Improvements are underway in locking down of the most egregious abuses of PUE. Three were recently summarized in Technical Scribblings RE Harmonizing Global Metrics for Data Center Energy Efficiency. In this report from John Stanley, the following were presented:
· Total energy to include all forms of energy whether electric or otherwise (e.g. gas fired chiller must include chemical energy being employed). I like it but It’ll be a challenge to implement
· Total energy should include lighting, cooling, and all support infrastructure. We already knew this but its worth clairifying since it’s a common “fudge” employed by smaller operators
· PUE energy should be calculated using source energy. This is energy at the source prior to high voltage distribution losses and including all losses in energy production. For example, for gas plants, it’s the fuel energy used including heat losses and other inefficiencies. This one seems hard to compute with precision and I’m not sure how I could possibly figure out source energy where some power is base load power and some is from peak plants and some is from out of state purchases. This recommendation seems a bit weird.
As with my recommendations in PUE and Total Power Usage Efficiency, these proposed changes add complexity while increasing precision. Mostly I think the increased complexity is warranted although the last, computing source energy, looks hard to do and I don’t fully buy that the complexity is justified.
It’s a good short read: Technical Scribblings RE Harmonizing Global Metrics for Data Center Energy Efficiency. Thanks to Vijay Rao of AMD for sending this one my way.
--jrh
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
Federal and state governments are prodigious information technology users. Federal Chief Information Security Office Vivek Kundra reports that the United States government is spending $76B annually on 10,000 different systems. In a recently released report, State of Public Sector Cloud Computing, Vivek Kundra summarizes the benefits of cloud computing:
There was a time when every household, town, farm or village had its own water well. Today, shared public utilities give us access to clean water by simply turning on the tap; cloud computing works in a similar fashion. Just like the water from the tap in your kitchen, cloud computing services can be turned on or off quickly as needed. Like at the water company, there is a team of dedicated professionals making sure the service provided is safe and available on a 24/7 basis. Best of all, when the tap isn’t on, not only are you saving water, but you aren’t paying for resources you don’t currently need.
§ Economical. Cloud computing is a pay-as-you-go approach to IT, in which a low initial investment is required to get going. Additional investment is incurred as system use increases and costs can decrease if usage decreases. In this way, cash flows better match total system cost.
§ Flexible. IT departments that anticipate fluctuations in user load do not have to scramble to secure additional hardware and software. With cloud computing, they can add and subtract capacity as its network load dictates, and pay only for what they use.
§ Rapid Implementation. Without the need to go through the procurement and certification processes, and with a near-limitless selection of services, tools, and features, cloud computing helps projects get off the ground in record time.
§ Consistent Service. Network outages can send an IT department scrambling for answers. Cloud computing can offer a higher level of service and reliability, and an immediate response to emergency situations.
§ Increased Effectiveness. Cloud computing frees the user from the finer details of IT system configuration and maintenance, enabling them to spend more time on mission-critical tasks and less time on IT operations and maintenance.
§ Energy Efficient. Because resources are pooled, each user community does not need to have its own dedicated IT infrastructure. Several groups can share computing resources, leading to higher utilization rates, fewer servers, and less energy consumption.
This document defines cloud computing and describes the federal government approach and then goes on to cover 30 case studies. The case studies are the most interesting part of the report in that they provide a sampling of the public sector move to cloud computing showing its real and project are underway and substantial progress is being made.
It’s good to see the federal government showing leadership at a time when the need for federal services are undiminished but the burgeoning federal deficit needs to be brought under control. The savings possible through cloud computing are substantial and the federal IT spending base is enormous, so its particularly good to be adopting this new technology delivery platform at scale.
· Document: State of Public Sector Cloud Computing
· Executive Summary: State of Public Sector Cloud Computing
Thanks to Werner Vogels for sending this article my way.
--jrh
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
Economic forces are more powerful than politics. Political change is slow. Changing laws takes time. Lobbyist water down the intended legislation. Companies find loop holes. The population as a whole, lacks the strength of conviction to make the tough decisions and stick with them.
Economic forces are far more powerful and certainly more responsive than political forces. Effectively, what I’m observing is great good can be done if there is a business model and profit encouraging it. Here’s my favorite two examples, partly because they are both doing great things and partly because they are so different in their approach, but still have the common thread of using the free market to improve the world.
Google RE<C
As a society, we can attempt to limit greenhouse gas emissions by trading carbon credits or passing laws attempting to force change but, in the end, it seems we just keep burning coal. In my view, the Google approach to tackling this problem is wonderful: invest in renewable energy technologies that can be cheaper than coal. More on the program: Google's Goal: Renewable Energy Cheaper than Coal and Plug into a Greener Grid: RE<C and RechargeIT Initiatives. They are working on solar thermal, high-altitude wind, and geo-thermal. The core idea is that, if renewable sources are cheaper than coal, economic forces would quickly make the right thing happen and we actually would stop burning coal. I love the approach but its fiendishly difficult.
Bill & Melinda Gates Foundation
Here’s a related approach. The problem set is totally different but there are some parallels with the previous example in that they are attempting to set up an economic system where it can be profitable to do good for society.
I attended a small presentation by Bill Gates about 5 years ago. By my measure, it was by far the best talk I’ve ever seen Gates gave. I suspect Bill wouldn’t agree that it was his best but it had a huge impact on me. No press was there and I saw nothing written about it afterwards, but two things caught my interest: 1) Gates’ understanding of world health problems is astoundingly deep, and 2) I loved his technique of applying free-market principles to battle world problems ranging from disease through population growth.
In this talk Bill noted that North American disease has a very profitable business model and consequently is heavily invested. Third world disease lacks a business model and, as a result, there is very little investment. It’s clear that many diseases are easy to control or even eradicate but there is no economic incentive and so there is no sustained progress. There are charity donations but no deep and sustained R&D investment since there are no obvious profits to be made. Bill proposed that we encourage business models that allows drug companies to invest R&D into third world health problems. They should be able to invest knowing they will be able make money on the outcome.
Current drug costs are driven almost exclusively by R&D costs. The manufacturing costs are quite low by comparison. Does this remind you of anything? It’s the software world all over again. So, the question that brings up is: Can we create a model where drugs are sold in huge volume at very low cost? I recall buying a copy of Unix for an IBM XT back in 1985 and it was $1,000 (Mark Williams Coherent). Today 1/10 of that will buy an O/S and many are free with the business model being built on services. Can we do the same thing to the drug world? Where else could this technique play out?
Using the free market to drive change is the most leveraged approach I’ve ever seen to drive change. Where else can we cost effectively change the economic model and drive a better outcome for society as a whole?
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
There have been times in past years when it really looked like we our industry was on track to supporting only a single relevant web browser. Clearly that’s not the case today. In a discussion with a co-working today on the importance of “other” browsers, I wanted to put some data on the table so I looked up the browser stats for this web site (http://mvdirona.com). I hadn’t looked for a while and found the distribution truly interesting:
Admittedly, those that visit this site clearly don’t represent the broader population well. Nonetheless, the numbers are super interesting. Firefox eclipsing Internet Explorer and by such a wide margin was surprising to me. You can’t see it in the data above but the IE share continues to decline. Chrome is already up to 17%.
Looking at the share data posted on Wikipedia (http://en.wikipedia.org/wiki/Usage_share_of_web_browsers#Summary_table and using the Net Market Share data) we see that IE has declined from over 91.4% to 61.4% in just 5 years. Again a surprisingly rapid change.
Focusing on client operating systems, from the skewed sample that accesses this site, we see several interesting trends: 1) Mac share continues to climb sharply at 16.6%, 2) Linux at 9%, 3) iphone, ipod and ipad in aggregate at over 5 ¼%, and 4) Android already over a ¼%.
Overall we are seeing more browser diversity, more O/S diversity, and unsurprisingly, more mobile devices.
--jrh
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
We live on a boat which has lots of upside but broadband connectivity isn’t one of them. As it turns out, our marina has WiFi but it is sufficiently unreliable that we needed another solution. I wish there was a Starbucks hotspot across the street – actually there is one within a block but we can’t quite pick up the signal even with an external antennae (Syrens).
WiFi would have been a nice solution but didn’t work so we decided to go with WiMAX. We have used ClearWire for over a year on the boat and, generally, it has worked acceptably well. Not nearly as fast as WiFi but better than 3G cellular. Recently ClearWire changed its name to Clear and “upgraded” the connectivity technology to full WiMAX. Unfortunately, the upgrade substantially reduced the coverage area, has been fairly unstable, and the Customer support although courteous and friendly is so far away from the engineering team that they basically just can’t make a difference no matter how hard they try.
We decided we had to find a different solution. I use AT&T 3G cellular with tethering and would have been fine with that as a solution. It’s a bit slower than Clear but its stable and coverage is very broad. Unfortunately, the “unlimited” plan we got some years ago is very limited to 5Gig/month and we move far more data than that. I can’t talk AT&T into offering a solution so, again, we needed something else.
Sprint now has a WiMAX service that offers good performance (although they can be a bit aggressive on throttling) and they have fairly broad coverage in our area and are expanding quickly (Sprint announces seven new WiMAX markets). Sprint has the additional nice feature on some modems where, if WiMAX is unavailable, it transparently falls back to 3G. The 3G service is still limited to 5Gig but, as long as we are on WiMAX a substantial portion of the month, we’re fine.
The remaining challenge was Virtual Private Networks (VPN) over WiMAX can be unstable. I really wish my work place supported Exchange RPC over HTTP (one of the coolest Outlook/Exchange features of all time). However, many companies believe that Exchange RPC over HTTP is insecure in that it doesn’t’ require 2 factor authentication. Ironically, many of these companies allow Blackberries’ and iPhones to access email without 2 factor auth. I won’t try to explain why one is unsafe and the other is fine but I think it might have something to do with the popularity of iPhones and Blackberries with execs and senior technical folks :-).
In the absence of RPC over HTTP, logging into the work network via VPN is the only answer. My work place uses Aventail but there are a million solutions out there. I’ve used many and love none. There are many reasons why these systems can be unstable, cause blue screens, and otherwise negatively impact the customer experience. But one that has been driving me especially nuts is frequent dropped connections and hangs when using the VPN over WiMAX. It appears to happen more frequently when there is more data in flight but to lose a connection every few minutes is quite common.
It turns out the problem is the default MTU on most client systems is 1500 but the WiMAX default is often smaller. It should still work and just be super inefficient but it doesn’t. For more details see http://www.amazon.com/Sierra-Wireless-Overdrive-Mobile-Hotspot/dp/B0032JTPMK.
To check Vista MTUs:
netsh interface ipv4 show subinterfaces
To change the MTU to 1400:
netsh interface ipv4 set subinterface "your vpn interface here" mtu=1400 store=persistent
I’m using an MTU of 1400 with Sprint and its working well. Thanks to Kitz.co.uk for the easy MTU update. If you are having flakey VPN support especially if running over WiMAX, check your MTU.
--jrh
James Hamilton, Amazon Web Services
1200, 12th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com
H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com
I'm on the technical program committe for ACM Science Cloud 2010. You should consider both submitting a paper and attending the conference. The conference will be held in Chicago on June21st, 2010 colocated with ACM HPDC 2010 (High Performance Distributed Computing).
The call for papers abstracst are due Feb 22 with final papers due March 1st: http://dsl.cs.uchicago.edu/ScienceCloud2010/
Workshop Overview:
The advent of computation can be compared, in terms of the breadth and depth of its impact on research and scholarship, to the invention of writing and the development of modern mathematics. Scientific Computing has already begun to change how science is done, enabling scientific breakthroughs through new kinds of experiments that would have been impossible only a decade ago. Today's science is generating datasets that are increasing exponentially in both complexity and volume, making their analysis, archival, and sharing one of the grand challenges of the 21st century. The support for data intensive computing is critical to advancing modern science as storage systems have experienced an increasing gap between their capacity and bandwidth by more than 10-fold over the last decade. There is an emerging need for advanced techniques to manipulate, visualize and interpret large datasets. Scientific computing involves a broad range of technologies, from high-performance computing (HPC) which is heavily focused on compute-intensive applications, high-throughput computing (HTC) which focuses on using many computing resources over long periods of time to accomplish its computational tasks, many-task computing (MTC) which aims to bridge the gap between HPC and HTC by focusing on using many resources over short periods of time, to data-intensive computing which is heavily focused on data distribution and harnessing data locality by scheduling of computations close to the data.
The 1st workshop on Scientific Cloud Computing (ScienceCloud) will provide the scientific community a dedicated forum for discussing new research, development, and deployment efforts in running these kinds of scientific computing workloads on Cloud Computing infrastructures. The ScienceCloud workshop will focus on the use of cloud-based technologies to meet new compute intensive and data intensive scientific challenges that are not well served by the current supercomputers, grids or commercial clouds. What architectural changes to the current cloud frameworks (hardware, operating systems, networking and/or programming models) are needed to support science? Dynamic information derived from remote instruments and coupled simulation and sensor ensembles are both important new science pathways and tremendous challenges for current HPC/HTC/MTC technologies. How can cloud technologies enable these new scientific approaches? How are scientists using clouds? Are there scientific HPC/HTC/MTC workloads that are suitable candidates to take advantage of emerging cloud computing resources with high efficiency? What benefits exist by adopting the cloud model, over clusters, grids, or supercomputers? What factors are limiting clouds use or would make them more usable/efficient?
This workshop encourages interaction and cross-pollination between those developing applications, algorithms, software, hardware and networking, emphasizing scientific computing for such cloud platforms. We believe the workshop will be an excellent place to help the community define the current state, determine future goals, and define architectures and services for future science clouds.
James Hamilton
e: jrh@mvdirona.com
w: http://www.mvdirona.com
b: http://blog.mvdirona.com / http://perspectives.mvdirona.com
Want to join a startup team within Amazon Web Services? I’m deeply involved and excited about this project and another couple of talented engineers could really make a difference. We are looking for:
User Interface Software Development Engineer
We are looking for an experienced engineer with a proven track record of building high quality, AJAX enabled websites. HTML, JavaScript, AJAX, and CSS experience is critical, along with Java and Tomcat. Experience with languages such as PHP, Perl, Ruby, Python, etc. is also useful. You must have significant experience in designing highly reliable and scalable distributed systems, including building front end website facing applications. You must thrive in a hyper-growth environment where priorities shift fast, have strong OO design and implementation experience, knowledge of web protocols, and in-depth knowledge of Linux tools and Java EE architectures.
For more information: https://us-amazon.icims.com/jobs/107700/job
Senior Software Development Engineer
We are looking for a Senior Software Engineer with a strong track record of building production scalable, high end, reliable, data driven distributed website systems. You must be able to tackle tough challenges and feel strongly not only about building good software but about making that software achieve its goals in an operational reality. You must thrive in a hyper-growth environment where priorities shift fast, have strong OO design and implementation experience, knowledge of web protocols, and in-depth knowledge of Linux tools and Java EE architectures.
For more information: https://us-amazon.icims.com/jobs/109479/job
If you are interested, send a resume to aws-jobinfo-lee@amazon.com. I’m looking forward to working with you.
--jrh
James Hamilton, Amazon Web Services
1200, 12th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com
H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com
I’ll be taking a brief hiatus from blogging during the first three weeks of August. Tomorrow we leave for China. You might wonder why we would go to China during the hottest time of the year. For example, our first stop, Xiamen, is expected to hit 95F today, which is fairly typical weather for this time of year (actually its comparable to the unusual weather we’ve been having in Seattle over the last week). The timing of the trip is driven by a boat we’re buying nearing completion in a Xiamen China boat yard: Boat Progress. The goal is to see the boat roughly 90% complete so we can catch any issues early and get them fixed before the boat leaves the yard. And, part of the adventure of building a boat, is to get a chance to visit the yard and see how they are built.
We love boating but, having software jobs, we end up working a lot. Consequently, the time we do get off, we spend boating between Olympia, Washington and Alaska. Since we seldom have the time for non-boat related travel, we figured we should take advantage of visiting China and see more than just the boat yard.
After the stop at the boat yard in Xiamen, we’ll visit Hong Kong, Guilin, Yangshou, Chengdu, and do a cruise of the Yangtze River and then travel to Xian followed by Beijing before returning home.
--jrh
James Hamilton, Amazon Web Services
1200, 12th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com
H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com
|