I’m long been a big fan of modular data centers using ISO standard Shipping containers as the component building block:
Containers have revolutionized shipping and are by far the cheapest way to move good over sea, land, rail or truck. I’ve seen them used to house telecommunications equipment, power generators, and even stores and apartments have been made using them: http://www.treehugger.com/files/2005/01/shipping_contai.php.
The datacenter-in-a-box approach to datacenter design is beginning to be deployed more widely with Lawrence Berkeley National Lab having taken delivery of a Sun Black Box and a “customer in eastern Washington” having taken delivery of a Rackable Ice Cube Module earlier this year.
Last summer I came across a book on Shipping Containers by Marc Levinson: The Box: How the Shipping Container Made the World Smaller and the World Economy Bigger. It’s a history of containers from the early experiments in 1956 through to mega-containers terminals distributed throughout the world. The book doesn’t talk about all the innovative applications of containers outside of shipping but does give an interesting background on their invention, evolution, and standardization.
-jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Earlier today I viewed Steve Jobs 2005 Commencement Speech at Stanford University. In this talk Jobs recounts three stories and ties them together with a common theme. The first was dropping out of Reed College and showing up for the courses he wanted to take rather than spend time on those he had to take. Dropping out was a tough decision but some of what he learned in these audited courses had a fundamental impact on the Mac and, in retrospect, appeared to be a good decision or at least one that lead to a good outcome. Getting fired from Apple was the second. Clearly not his choice, not what he would have wanted to happen but it lead to Pixar, Next and rejoining Apple stronger and more experienced than before. Again, a tough path but one that may have lead to a better overall outcome. Likely he is a better and more capable leader for the experience. Finally, facing death. Death awaits us all and, when facing death, it becomes clear what really matters. It becomes clear that following your heart is what is really important. Don’t be trapped by Dogma, don’t live other people’s lives, and have the courage to follow your own intuition. Clearly nobody wants to approach to death but knowing it is coming can free each of us to realize we can’t hide, we don’t have forever, and those things that scare us most are really tiny and insignificant when compared with death. Facing death can free us to take chances and to do what is truly important even if success looks uncertain or the risk is high.
The theme that wove these three stories together and Jobs parting words for his listeners was to “stay hungry and stay foolish”.
It’s a good read: http://news-service.stanford.edu/news/2005/june15/jobs-061505.html. Or you can view it at: http://www.youtube.com/watch?v=D1R-jKKp3NA.
Sent my way by Michael Starbird-Valentine.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Past experience suggests that disk and memory are the most common server component failures but what about power supplies and mother boards? Amaya Souarez of Global Foundation Services pulled the data on component replacements for the last six months of 2007 and we saw this distribution:
1. Disks: 59.0%
2. Memory: 23.1%
3. Disk Controller: 05.0% (Fiber HBAs and array controllers)
4. Power Supply: 03.4%
5. Fan: 01.1%
6. NIC: 01.0%
After disk and memory the numbers fade to the noise fairly quickly. Not a big surprise. What I did find quite surprising was the percentage of systems requiring service. Some systems will require service more than once and some systems will have multiple components replaced in a single service. Ignoring these factors and treating each logged component replacement as a service event, in the sample we looked at, we found we had 192 service events per 1,000 servers in six months. Making the reasonable assumption that this data is not sensitive to the time of year, that would be 384 service events per 1,000.
The good news is that server service is fairly inexpensive. Nonetheless, these data reinforce the argument I’ve been making for the last couple of years: the service-free, fail-in-place model is where we should be going over the longer term. I wrote this up in http://research.microsoft.com/~jamesrh/TalksAndPapers/JamesRH_CIDR.doc but the basic observation is that the cost per server continues to decline while people costs don’t.
Going to a service free-model can save service costs but, even more interesting, in this model the servers can be packaged in designs optimized for cooling efficiency without regard to human access requirements. If technicians need to go into a space, then the space needs be safe for humans and meet multiple security regulations, a growing concern, and there needs to be space for them. I believe we will get to a model where servers are treated like flash memory blocks: you have a few thousand in a service-free module, over time some fail and are shut off, and the overall module capacity diminishes over time. When server work done/watt improves sufficiently or when the module capacity degrades far enough, the entire module is replaced and returned to the manufacturer for recycling.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Rules of thumb help us understand complex systems at a high level. Examples are that high performance server disks will do roughly 180 IOPS, or that enterprise system administrators can manage roughly 100 systems. These numbers ignore important differences between workloads and therefore can’t be precise, but they serve as a quick check. They ignore, for example, that web servers are MUCH easier to administer than database servers. Whenever I’m looking at a new technique, algorithm, or approach, I always start with the relevant rules of thumb and note the matches and differences. Where it differs, I look deeper. Sometimes I find great innovation and learn that the rules need to be updated to take into account the efficiencies of this new approach. But, more frequently, I find an error in the data, incorrect measurement technique, an algorithm that only works over a narrow set of workloads, or other restriction. Basically, a good repertoire of rules of thumb is useful in helping to find innovation and quickly spot mistakes.
Everyone does this at some level, although often they aren’t using formal rules but more of an informal gut feel. This gut feel helps people avoid mistakes and move more quickly without having to stare at each new idea and understand it from first principles. But there is a danger. Every so often, the rules change and if you don’t update your “gut feel” you’ll miss opportunities and new innovations.
Over the years, I’ve noticed that the duration from the first breakthrough idea on a topic to it actually making sense and having broad applicability is 7 to 10 years. The earliest research work is usually looking beyond current conditions and when the ideas are first published, we usually don’t know how to employ them, don’t yet have efficient algorithms, or the breadth of problems solved by the new ideas is not yet sufficiently broad. It usually takes 7 to 10 years to refine and generalize an idea to the point where it is correct and broadly useable.
Now you would think that once an idea “makes sense” broadly, once it’s been through it’s 7 to 10 years of exile, it would be ready for broad deployment. Ironically, there is yet one more delay and this one has been the death of many startups and a great many development projects. Once an idea is clearly “correct”, applicable, and broadly generalized, enterprise customers still typically won’t buy it for 5 to 7 years. What happens is that the new idea, product, or approach violates their rules of thumb and they simply won’t buy it until the evidence builds over time and they begin to understand that the rules have changed. Some examples:
· Large memories: In the early ’90s it became trivially true that very large memories were the right answer for server-side workloads, especially database workloads. The combination of large SMPs and rapidly increasing processor performance coupled with lagging I/O performance and falling memory prices made memory a bargain. You couldn’t afford to not buy large memories and yet many customers I was working with at the time were much more comfortable buying more disk and more CPU even though it was MORE expensive and less effective than adding memory. They were trapped in their old rules of thumb on memory cost vs value.
· Large SMPs: In the late ’80s customers were still spending huge sums of money buying very high end water cooled ECL mainframes when they should have been buying the emerging commodity UNIX SMPs and saving a fortune. It takes a while for customers and the market as a whole to move between technologies.
· Large clusters: in the late ’90s and to a lesser extent even to this day, customers often buy very large SMPs when they should be buying large clusters of commodity servers. Sure, they have to rewrite software systems to run in this environment but, even with those costs, in large deployments, mammoth savings are possible. It’ll take time before they are comfortable and it’ll take time before they have software that’ll run in the new environment.
Basically, once an idea becomes “true” it still has 5 to 7 more years before it’s actually in broad use. What do we learn from this? First, it’s very easy to be early and to jump on an idea before it’s time. The dot com era was full of failures even though the same idea will actually succeed in the hands of a new startup over the next couple of years (5 to 7 year delay). It’s hard to have the right amount of patience and it’s very hard to sell new ideas when they violate the current commonly held rules of thumb. The second thing we learn is to check our rules of thumb more frequently and to get good at challenging the rules of thumb or gut feels of others when trying to get new ideas adopted. Understand that some of our rules might no longer be true and that some of the rules of thumb used by the person you are speaking with may be outdated.
Here are four rules of thumb, all of which were inarguably true at one point in time, and each is either absolutely not true today or on the way to being broken:
· Compression is a lose in an OLTP system: This is a good place to start since compression is a clear and obvious win today. Back in 1990 or thereabout I argued strongly against adding compression to DB2 UDB (where I was lead architect at the time). At the time, a well tuned OLTP system was CPU bound and sufficient I/O devices had been added to max out the CPU. The valuable resource was CPU in that you could always add more disk (at a cost) but you can’t just add more CPUs. At the time, 4-way to 8-way systems were BIG and CPUs were 100x slower than they are today. Under those conditions, it would be absolutely nuts to trade off CPU for a reduction in I/O costs for the vast majority of workloads. Effectively we would be getting help with a solvable problem and, in return, accepting more of an unsolvable problem. I was dead against it at the time and we didn’t do it then. Today, compression is so obviously the right answer it would be nuts not to do it. Most very large scale services are running their OLTP systems over clusters of many databases servers. They have CPU cycles to burn but I/O is what they are short of and where the costs are. Any trick that can reduce I/O consumption is worth considering and compression is an obvious win. In fact, compression is now making sense higher up the memory hierarchy and there are times when it makes sense to leave data compressed in memory only decompressing when needed rather than when first brought in from disk. Compression is obviously a win in high end OLTP systems and beyond and, as more customers move to clusters and multi-core systems this will be just keep getting more clear.
· Bottlenecks are either I/O, CPU, or inter-dispatch unit contention: I love performance problems and have long used the magic three as a way of getting a high level understanding of what’s wrong with a complex system that is performing poorly. The first thing I try to understand when looking at a poorly performing system is whether the system is CPU bound, I/O bound (network, disk, or UI if there is one), or contention bound (processes or threads blocked on other processes or threads – basically, excess serialization). Looking at these three has always been a gross simplification but it’s been a workable and useful simplification and has many times guided me to the source of the problem quickly. Increasingly, these three are inadequate in describing what’s wrong with a slow system in that memory bandwidth/contention is increasingly becoming the blocker. Now in truth, memory bandwidth has always been a factor but it typically wasn’t the biggest problem. Cache conscious algorithms attempt to solve the memory bandwidth/contention problem but how do you measure whether you have a problem or not? How do you know if the memory and/or cache hierarchy is the primary blocking factor?
The simple answer is actually very near to the truth but not that helpful: it probably IS the biggest problem. Memory bandwidth/contention is becoming at least the number two problem for many apps only behind I/O contention. And, for many, it’s the number one performance issue. How do we measure it quickly and easily and understand the magnitude of the problem? I use cycles per instructions as an approximation. CPUs are mostly super-scalar which means they can retire (complete) more than one instruction per clock cycle. What I like to look at is rate that instructions are retired. This gives a rough view of how much time is being wasted in pipeline stalls most of which are caused by waiting on memory. As an example to give a view for the magnitude of this problem, note that most CPUs I’ve worked with over the last 15 years can execute more than one instruction per cycle and yet I’ve NEVER seen it happen running a data-intensive, server-side workload. Good database systems will run in the 2.0 to 2.5 cycles per instruction (CPI) and I’ve seen operating system traces that were as bad as 7.5 cycles per instruction. Instead of executing multiple instructions per cycle, we are typically only executing fractions of instructions each cycle. So, another rule of thumb is changing: you now need to look at the CPI of your application in addition to the big three if you care deeply about performance.
· CPU cycles are precious: This has always been true and is still true today but it’s become less true fast. 10 years ago, most systems I used, whether clients or servers, were CPU bound. It’s rare to find a CPU bound client machine these days. Just about every laptop in the world is now I/O bound. Servers are quickly going the same route and many are already there. CPU performance increases much faster than I/O performance so this imbalance will continue to worsen. As the number of cores per socket climbs, this imbalance will climb at an accelerated pace. We talked above about compression making sense today. That’s because CPU cycles are no longer the precious resource. Instead of worrying about path length, we should be spending most of our time reducing I/O and memory references. CPU cycles are typically not the precious resource and multi-core will accelerate the devaluation of CPU resources over memory and I/O resources.
· Virtual Machines can’t be used in services: I chose this as the last example as it is something that I’ve said in the last 12 months and yet it’s becoming increasingly clear that this won’t stay true for long. It’s worth looking more closely at this one. First, the argument behind virtual machines not making sense in the services world is this: when you are running 10s to 1000s of systems of a given personality, why mix personalities on the same box? Just write the appropriate image to as many systems as you need and run only a single application personality per server. To get the advantages of dynamic resource management often associated with using virtual machines, take a system running one server personality and re-image it to another. It’s easy to move the resources between roles. In effect you get all the advantages without the performance penalty of virtualization. And, by the way, it’s the virtualization penalty that really drives this point home. I/O intensive workloads often run as badly as 1/5 the speed (20% of native) when running in a virtual machine and ½ the throughput is a common outcome. The simple truth is that the benefits of using VMs in high scale services are too easy to obtain using other not quite as elegant techniques and the VM overhead at scale is currently unaffordable. Who could afford to take a 20,000 system cluster and, post virtualization, have to double the server count to 40,000? It’s simply too large a bill to pay today. What would cause this to ever become interesting? Why would we ever run virtual machines in a service environment?
At a factor of two performance penalty, it’s unlikely to make sense in large services (hardware is the number one cost in services -- well ahead of all others – I’ll post the detailed breakdown sometime) so doubling this comes to close to adding 25% to the overall service cost. Unacceptable. When do they make sense is:
1. When running non-trusted code, some form of isolation layer is needed. ASP and the .Net Common Language Runtime are viable isolation boundaries but, for running arbitrary code, VMs are an excellent choice. That’s what Amazon EC2 uses for isolation. Another scenario used by many customers is to run lightweight clients and centrally host the client side systems in the data center. For these types of scenarios, VMs are a great answer.
2. When running very small services. Large services need 100s to 10s of thousands of systems so sharing servers isn’t very interesting. However, for every megaservice, there are many very small services some of which could profit from server resource sharing.
3. Hardware independence. One nice way to get real hardware independence and to enforce it is to go with a VM. I wouldn’t pay a factor of two overhead to get that benefit but, for 10% it probably would make sense. You could pay for that overhead but reduction in operational complexity which lowers costs, brings increased reliability, and allows more flexible and effective use of hardware resources.
As a final observation in support of VMs in the longer term, I note that the resource they are wasting in largest quantity today is the resource that is about to be the most plentiful looking forward. Multi-core, many core, and the continuing separation of compute performance vs I/O performance, will make VMs a great decision and a big part of the future service landscape.
This last example is perhaps the most interesting example of a changing rule of thumb in that it’s one that is still true today. In mega-deployments, VMs aren’t worth the cost. However, it looks like this is very unlikely to stay true. As VM overhead is reduced and the value of the squandered CPU resources continues to fall, VMs will look increasingly practical in the service world. It’s a great example of a rule of thumb that is about to be repealed.
Rules of thumb help us quickly get a read on a new idea, algorithm or approach. They are a great way to quickly check a new idea for reasonableness. But, they don’t stay true forever. Ratios change over time. Make sure you are re-checking your rules every year or so and, when selling new ideas, be aware the person you are talking to may be operating under an outdated set of rules. Bring the data to show which no longer apply, or you’ll be working harder than you need to.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
When you look at disk transfer rates, it’s pretty obvious that the faster you spin them, the lower the rotational latency and the better the transfer rates. It’s also very clear that disk transfer rates are improving much slower than memory subsystem bandwidth. Why not rotate disks faster? We’ve had 15,000 RPM enterprise disk for a considerably long time. Why have there been no recent improvements on this dimension?
The reason I’ve long understood is that power consumption scales cubically with the RPM but there is more to it than that. I recently came across a claim that power scaled to the fifth power with RPM. That didn’t seem right so I decided to go check the facts. I asked Dave Anderson of Seagate Research if he could help chase down the real data. By the way, if you ever get a chance to see Dave speak, jump on it. He knows an enormous amount about all aspects of spinning media technology and, when he really get down to the details, it’s amazing that disk can work at all. At the outside edge of a 15,000 RPM drive, the platter is going by the head at 118 MPH, the head is flying at 0.5 microinches (less than the wavelength of visible light), the track width is less than 1/300 the width of a page, nearby vibration sources such as other disks mean the platter will be vibrating slightly and there is always some non-repeatable runout (NRRO). As the tracks shoots by the disk, the blocks need to be read, decoded, identified to ensure the right block was read, and error checked so significant computational work needs to be done as well. Phenomenal technology.
Disks really are amazing but returning to why RPMs haven’t been increasing of late, I checked with Dave Anderson a senior engineer at Seagate. The short answer is cost. As RPM is increased, NRRO increases quickly, as does vibration both leading to serious tracking complexity. The cost of the drive motors goes up, air drag increased dramatically, as does power consumption. What’s exact relationship for disk power consumption as a function of RPM? From Dave:
There are 2 general cases for power consumption today, a cube relation for the general case, and a square relation, where the discs are packed so closely together, they approximate a cylinder.
Apparently there can still be a **4 condition where there is extensive turbulence in a single disk drive.
Power consumption going up cubically is problematic but enterprise disk today only draw 15 to 18W each. I would never recommend wasting power but many deployments need to use more disks to get the performance they need. In effect the power is often getting spent to support the given workload anyway. Would less power be consumed by fewer, faster disks? Further complicating the equation and making that gain harder to find, rotational latency improvements diminish rapidly as RPM goes up. The improvement from the current high of 15,000 RPM to say 20,000 RPM isn’t very significant:
.jpg)
Dave Anderson Chart
I suspect we will see increased RPM in the future but increase in cost and power coupled with diminishing returns and far increased complexity from all the factors listed above suggest we’ll not see the RPM going up very quickly in the future. Some speculate, we’ve seen the last RPM increase but I’ve never been a subscriber of hit-the-wall theory. We always eventually find a way.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
In past blog entries, I’ve talked of the impact of Flash on server-side systems. On the client, flash SSDs can help with at least two different markets paradoxically at different ends of the cost spectrum: 1) economy low cost laptops, and 2) high performance laptops. At the low end, flash can help make less expensive systems in that hard disk drives are mechanical devices with motors and actuators and they have a price floor. Even very small disks need to have a motor and an actuator so getting a HDD for much less than $50 is quite difficult. For very low cost devices with very small storage requirements, a flash SSD can be cheaper than a disk of similar size. And, in addition to being cheaper than HDDs in very small form factors, flash SSDs also consume less power, are more durable, and can operate reliably in broader environmental conditions. Perhaps the prototypical inexpensive laptop is the, One Laptop Per Child project. It uses NAND flash for persistent storage: http://wiki.laptop.org/index.php/Hardware_specification.
On the other end of the spectrum, high-end laptops where performance, light-weight, silence, and long battery life are all important factors, NAND flash SSDs again are becoming common. Many high-end laptops are shipped with flash SSDs rather than depending on a HDD. Some examples: Samsung, Sony Vaio, Dell Lattitude, HP Compaq, Asus, and many others.
Flash SSDs are also emerging as a common choice in ruggedized laptops due to the broad environmental range within which flash SSDs operate reliably. They are also becoming a common choice in Ultra Mobile PCs such as the Samsung Q1.
Flash SSDs are on track to be used in a broad percentage of the laptop market. EE times, for example, estimates that flash SSDs will be the supplied storage media in 38% of laptop market:
The above is from: http://www.eetimes.com/showArticle.jhtml?articleID=204400359 (Jack Creasey sent the article my way). Looking at specifically corporate laptops, I would expect the penetration to be far higher than 38%.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
The internet was designed in a different time at a different scale. It’s rare that a design continues to work at all when scaled multiple orders of magnitude so it remains impressive but there are issues. The blackholing of YouTube over the weekend showed one of them. Routing is fragile and open to administrative error and also certain forms of attack but this particular example was the more common one: human error.
Over the weekend, a decision was made in Pakistan took down Youtube for two hours. Here’s what happened. Pakistan Telecom received a government order to block Pakistani access to YouTube. They did this for their network (most of Pakistan) but also advertised this route incorrect route to their provider, PCCW, as well. PCCW shouldn’t have accepted iy but did. Since PCCW is big and hence fairly credible, the error propagated quickly throughout the world from there.
This issue was inconvenient but the same sort of attack can be constructed intentionally to disrupt access to a web sites or to direct users to a web site masquerading as another.
Perhaps the best detail is on the Renesys blog: http://www.renesys.com/blog/2008/02/pakistan_hijacks_youtube_1.shtml. Other sources: http://www.news.com/8301-10784_3-9878655-7.html, http://www.nytimes.com/2008/02/26/technology/26tube.html?_r=1&ref=business&oref=slogin.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
I get several hundred emails a day, some absolutely vital and needing prompt action and some about the closest thing to corporate spam. I know I’m not alone. I’ve developed my own systems on managing the traffic load and, on different days, have varying degrees of success in sticking to my systems. In my view, it’s important not to confuse “processing email” with what we actually get paid to do. Email is often the delivery vehicle for work needing to be done and work that has been done, but email isn’t what we “do”.
We all need to find ways of coping with all the email while still getting real work done and having a shot at a life outside of work. My approach is fairly simple:
· Don’t process email in real time or it’ll become your job. When I’m super busy, I process email twice a day: early in the morning and again in the evening. When I’m less heavily booked, I’ll try to process email in micro bursts rather than in real time. It’s more efficient and allows more time to focus on other things.
· Shut off email arrival sounds and the “new mail” toast or you’ll end up with 100 interruptions an hour and get nothing done but email.
· I get up early and try to get my email down to under 10 each morning. I typically fail but get close. And I hold firm on that number once a week. Each weekend I do get down to less than 10 messages. If I enter the weekend with hundreds of email items, I get to work all weekend. This is a great motivator to not take a huge number of unprocessed email messages into the weekend.
· Do everything you can to process a message fully in one touch. I work hard to process email once. As I work through it, I delete or respond to everything I can quickly. Those that really do require more work I divide into two groups: 1) those I will do today or, at the very latest by end of week, I flag with a priority and leave in my inbox for processing later in the day (many argue these should be moved to a separate folder and they may be right). The longer-lived items go into my todo list and I remove them from my inbox. Because I get my email down to under 10 each week and spend as much of my weekend as needed to do this, I’m VERY motivated to not have many emails hanging around waiting to be processed. Consequently most email is handled up front as I see them and the big things are moved to the todo list. Very few are prioritized for handling later in the day.
· I chose not to use rules to auto-file email. Primarily I found that if I sent email directly to another folder, I almost never looked at it. So I let everything come into my inbox and I deal with them very quickly and, for the vast majority, they will only be touched once. If I really don’t even want to see them once, I just don’t subscribe or ask not to get them.
· Set your draft folder to be your inbox. With email systems that use a separate folder for unsent mail, there is risk that you’ll get a message 90% written and ready to be sent, get interrupted and then forget to send it. I set my draft folder to be my inbox so I don’t lose unsent email. Since my email is worked down to under 10 daily, I’ll find it there for sure before end of day.
· Don’t bother with complicated folder hierarchies—they are time-consuming to manage. If you want to save something, save it in a single folder or simple folder hierarchy and let desktop search find it when you need it. Don’t waste time filing in complex ways.
· Finally, be realistic: if you can’t process at the incoming rate, it’ll just keep backing up indefinitely. If you aren’t REALLY going to read it, then delete it or file it on the first touch. Filing it has some value in that, should you start to care more in the future, you can find it via full text search and read it then.
Jeff Johnson of MSN pointed out this excellent talk on email management called “Inbox Zero” by Merlin Mann: http://www.43folders.com/2007/07/25/merlins-inbox-zero-talk. Merlin’s advice is good and he presents well.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Yesterday, Data Center knowledge reported that Sun was working on a Cloud Platform to compete with Amazon AWS: Project Caroline. The data behind the report comes from a upcoming Java One 2008 presentation by Sun Distinguished Engineer, Bob Scheifler. The talk announcement and synopsis is posted at: http://research.sun.com/projects/caroline/ and, even better, the slides are already up at: http://developers.sun.com/learning/javaoneonline/2007/pdf/TS-1991.pdf.
The full functionality supported by Caroline is actually beyond that offered by Amazon AWS. Included is:
· Virtualizes key resources such as network and compute and provides horizontally scaled pool for each
o Programmatic control of resource allocation, increasing or decreasing without human interaction,
· Java VMs (rather than offer fully general Virtual Machines as Amazon does with EC2, the Java APIs are offered as the only programming abstraction)
· Identity provider
· Eclipse based dev tools
· ZFS file system with storage reservation, access controls, snapshots (with rollback) and quotas
· Database (PostgreSQL)
· Networking: VLAN control, VPN support, dynamic NAT, L4 and L7 load balancing, DNS config
Overall it looks pretty interesting – the concepts all look good. The true test and the measure of whether this will actually be an AWS competitors won’t come until the service is made available at scale. Nonetheless, its great seeing more pay-as-you-go service offerings coming available.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Yet another argument in favor of Degraded Operations Mode (http://mvdirona.com/jrh/perspectives/2008/01/22/DegradedOperationsMode.aspx) emerged last week. All of Amazon AWS (S3, SimpleDB, Simple Queuing Service, EC2, etc.) down for several hours last week: http://mvdirona.com/jrh/perspectives/2008/02/15/DowntimeAmazonS3SimpleDBSQS.aspx. The outage was reportedly due to a authentication storm: http://www.highscalability.com/s3-failed-because-authentication-overload (Mike Neil sent this my way).
Remember, you’ll never have the capacity for the biggest load inrush and, no matter how hard you try, your capacity planning will continue to only slightly better than the weather report for next week. When you don’t know what’s coming, design systems to operate through adversity: Degraded Operations Mode.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
I recently was in a meeting with several physicians. One of them reported a result I have always suspected, but at a magnitude I never would have guessed. The core observation was that that 80% of medical diagnoses were incorrect. The other doctors in the room confirmed this number to be roughly consistent with their experience. A less anecdotal support for this estimated high error rate is found in those cases where a “gold standard” diagnostic test is discovered where there previously wasn’t one. What has been found in many of these cases is that upwards of 80% of the previous diagnoses were incorrect. Several examples were given from different disease populations where a gold standard test has emerged.
How could one of the best funded medical systems in the world possibly be misdiagnosing so many patients? The speculation was that it’s a combination of two factors: 1) doctors have VERY little information on the patient, often having never seen them before and, if they have met in the past, it is usually only for an hour or so a year, and 2) insufficient diagnostic information is available. Tests take time, cost money, sometimes are misapplied (e.g. poor X-rays), and some medical issues lack affordable, and highly reliable tests.
At first glance this incredible inaccuracy is shocking and hard to accept but, upon reflection, I have seen similar problems in my distant past as a professional auto mechanic. Misdiagnosis and incorrect parts replacement is common. Repeat, returning, and unsolved problems are not uncommon. Automobiles are complex systems, but much less complex than human beings, so its believable that medicine sees the same problems in a more exaggerated form. In the automotive world, expensive misdiagnoses are battled on two fronts. This first is through high quality data acquisition and diagnostic equipment to pinpoint the problem. The second approach is to move from a repair model to a parts replacement model. The size of the replaceable component is increasing, which both minimizes labor costs and reduces the likelihood of error (replacing large complex components as a whole normally succeeds at the cost of some wastage). This second technique doesn’t apply well to the medical world but the former does: collect massive amounts of information to improve diagnostic success rates.
I get two things out of this discussion: 1) taking an active role in the collection and management of your medical records is worth the investment, 2) be better informed (just as knowing a bit about a car can help you communicate symptoms to auto-mechanic and the same is true with medical issues), and 3) well-executed, diagnostic tests are the most important part of any diagnosis. In fact, well executed diagnostic tests can be more important than the skill and experience level of the diagnosing physician.
We’ve recently announced HealthVault (http://www.healthvault.com/), a site supporting 1) health related content and search, 2) a central data storage system for health related information, and 3) connectivity to monitoring devices such as blood glucose readers, blood pressure monitors, heart rate monitors. More information on HealthVault is at: HealthBlog. This is early stage work but the combination of a central data repository and automated health information gathering has huge potential. Technical, social, and legal issues must be overcome to realize the full potential of this service, but if we found a way to directly acquire diagnostic data from hospitals and clinics, this service could become truly amazing.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-C/1279, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
If you run a big service and claim to have never had down time you either 1) have close to zero customers or 2) are lying. It’s almost that simple.
There is considerable concern that Amazons AWS service was down for several hours:
· http://www.roughtype.com/archives/2008/02/amazons_s3_util.php
· http://gigaom.com/2008/02/15/amazon-s3-service-goes-down/
· http://www.centernetworks.com/amazon-s3-down-error
Thanks to Jeff Currier and Soumitra Sengupta who told me about the downtime as it was happening last week. The service was reported to be down at 4:30AM. At 10:17, they reported it was resolved. There are a couple of lessons in here but the first is that internal IT goes down, high scale services go down, client systems fail, networks stop operating, power failures happen, etc. That’s just the way it is. You can spend to reduce these factors and you can try to take complete control of the IT infrastructure to avoid them impacting you. Ironically, in my experience, those that take over and run the entire infrastructure typically do it at lower scale with less experience and have downtime as well. These small scale services end up costing much more and yet deliver very little additional uptime. You read about commodity priced, high scale services when they go down. For example, RIM was down last week. But, the good ones really don’t go down that frequently. High scale, commodity infrastructure is actually pretty solid and compares very well to vertical, control-all-aspects-of-the-IT-infrastructure approaches. Amazon AWS generally has earned a pretty good reliability record.
The second lesson is perhaps the hardest to learn and the most important: customers need information. If a service goes down – actually, I should say, when a service goes down – you need to tell customers what is happening and set expectations on service restoration right away. There is a temptation to hide the facts because, well, downtime is embarrassing. Hiding it simply doesn’t work. When people don’t know what is happening, they assume the worst and think you are trying to hide something or aren’t responding properly. Tell them what is happening, invest resources in keeping them up to date with progress, and tell them when you expect to be back up.
It’s hard, it’s embarrassing, but this one matters more than any other. Long after the downtime is forgotten, people will remember how you handled it. Transparency wins when it comes to service operation – customers who have decided to bet there jobs on your service need timely information for their customers. If you embarrass your customers, they remember forever. A little downtime is unfortunate and you need to be getting better all the time but that’s forgivable. Just get them the information they need for their dependent businesses.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Google has published an interesting study of Mobile Search trends (sent my way by Tren Griffin). In this study the authors looked at over 1M queries submitted to Google Mobile web search over the course of a one month period. They found that the average search query was 2.56 words. (This is surprisingly similar to the average desktop query at 2.6 and the average PDA query at 2.35). They predictably found a uniform relationship between query length in characters and the length of time it took to enter it. The average query took 44.8 seconds including network interactions. They estimate the overhead to be roughly 5 seconds, meaning the user is willing to spend nearly 39 seconds entering a query. This is amazingly high. It gives an idea of how valuable the query results are if users are willing to take that long to enter it. The researchers found less query diversity in the mobile world than the desktop world. The mobile click-through rate on queries was over 50%.
I also found it interesting that users are entering queries faster this year than the comparative data from 2005. The average query time fell from 66.3 seconds to 44.8 seconds (including communications overhead). The paper speculates this is a combination of improved keyboards and a population more comfortable with using devices.
The full paper is available from: http://www.maryamkamvar.com/publications/KamvarBalujaComputerMagazine.pdf.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-C/1279, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
I upgraded my Samsung SGH-i607 to Windows Mobile 6 earlier today. I had held off upgrading until today having been told that Internet Connection Sharing doesn’t work on the AT&T Windows Mobile 6 build. Actually, it’s even a bit of a hassle to make it work on Win Mobile 5 but it can be done on both WM5 and 6. ICS isn’t actually removed from the WM6 build, it’s just not exposed in the user interface as was done with WM5 and, in WM6, there are security settings preventing it from operating. So it is more work to enable it but not really all that much (details on the page referenced below).
Overall I’m happy with the upgrade. I’ve added to my Blackjack Hack, Tip, Techniques & Utilities page to include WM6 installation instructions, application unlocking instructions, pointer to the Internet Connection Sharing enabling procedure, instruction on how to move email and IE temp files to the storage card, a higher information density home page, and a few utilities. If you have accumulated other interesting tricks, send them my way. For example, I’ve not yet managed to SIM unlock this one.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
A few months back I was in a debate about the value of shared code segments between virtual machines. In my view there is no question that shared code across VMs has some value but code is small compared to data so the impact will be visible but not fundamental. What follows is an inventory of a typical client-side systems.
This experiment was done on an IBM T43 laptop with 1GB of memory running Vista RTM, desktop search, Foldershare (it rocks), and Outlook. Outlook was in use prior to and during the measurement. The system has been running for three days since the last boot. The summary stats are:
|
Classification |
pages |
Meg |
% |
|
|
|
|
|
Kernel: |
65824 |
257.125 |
25% |
|
User: |
195913 |
765.2852 |
75% |
|
Total: |
261737 |
1022.41 |
|
|
|
|
|
|
Kernel Pages |
|
|
|
|
Kernel Image: |
7395 |
28.88672 |
11% |
|
Kernel Pure Data: |
58429 |
228.2383 |
89% |
|
Kernel Total: |
65824 |
257.125 |
|
|
|
|
|
|
User Pages |
|
|
|
|
User Code: |
32348 |
126.3594 |
17% |
|
User Data: |
163565 |
638.9258 |
83% |
|
User Total: |
195913 |
765.2852 |
|
Immediately after boot, 22% of the memory was code which makes sense. As the O/S and apps come up, all constructors and initializers run. After being memory resident for a few days, only those pages currently in use stay loaded and the user code percentage fell to 17%. Ironically, code load time is an issue at start-up time but the actually percentage of code resident in memory over longer runs is fairly small. Vista Superfetch helps with the code load times but, from looking at this data It’s clear that flash memory could make a huge difference to O/S boot and application load times.
The percentage of memory holding code pages is not that high so when going after memory bloat, look first to the data.
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
Yesterday, Intel and Micron announced a generational step forward in NAND Flash Write I/O performance. From the Intel Press release:
The new high speed NAND can reach speeds up to 200 megabytes per second (MB/s) for reading data and 100 MB/s for writing data, achieved by leveraging the new ONFI 2.0 specification and a four-plane architecture with higher clock speeds. In comparison, conventional single level cell NAND is limited to 40 MB/s for reading data and less than 20 MB/s for writing data.
They don’t actually say it’s an SLC device but they compare it to SLC and it has the typical wear characteristics of SLC (100,000 cycles). More data from the Micron web site:
|
Features |
Benefits |
|
Density |
8Gb–16Gb |
Industry-standard densities |
|
Performance |
200 MB/s Sustained READ
100 MB/s Sustained WRITE
1.5ms (TYP) Erase Performance |
Delivers the fastest read and write throughputs ever for a NAND Flash device |
|
Endurance (cycles) |
100,000 |
High-endurance enables applications that require intensive program and erase operation while prolonging memory life |
|
Interface |
Async/Sync
ONFI 1.0/2.0 |
Standard interface enables a high degree of interoperability |
|
Temperature Range |
−25˚C to +85˚C |
Wide temperature range is ideal for rugged environments |
|
Configuration |
1.8V, x8 |
Industry-standard configuration enables easy system design |
|
Package |
100-ball BGA |
Industry-standard packaging enables easier density migration |
Expect shipments in the latter half of 2008. We should start seeing interesting applications of this technology in SSDs and other devices this year.
Intel Press Release: http://www.intel.com/pressroom/archive/releases/20080201corp.htm
More data from Micron: http://www.micron.com/products/nand/high_speed/index
--jrh
James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com
H:mvdirona.com | W:research.microsoft.com/~jamesrh
|