Friday, November 07, 2014

In the Amazon Web Services world, this has always been a busy time of the year. Busy, because, although we aim for a  fairly even pace of new service announcements and new feature releases all year, invariably, somewhat more happens towards the end of the year than early on. And, busy, because the annual AWS re:Invent conference is in early November and this is an important time to roll out new services or important features. This year is no exception and, more than ever, there is a lot to announce at the conference. It should be fun.

I enjoy re:Invent because it’s a chance to talk to customers in more detail about what we have been building, learn how they are using it, and what we could do to make the services better. It’s always exciting to see what customer have done with AWS over the last year and I like the fact that customer presentations are a big part of the conference. It’s great to hear from an engineer how a service should work but almost always more interesting to see how customers are actually using the service. Each year there are more customers who have decided to move aggressively and go 100% cloud hosted and there are more and more stories of large, mission critical applications that have been moved to the cloud. It’s good to hear the details behind these decisions and how they were executed.


Andy Jassy who leads AWS will give the conference keynote Wednesday November 12th at 9am. If you can’t be at re:Invent in person, it will also be available via live streaming. This session is always packed with new service and major feature announcements so, if you can only watch one session, this is it.


Later that day at 4:30, I’ll be presenting AWS Innovation at Scale which will also be available via video streaming. In this session, I’ll be putting some quantitative numbers on current AWS growth rates. Then I’ll focus on two main areas: networking and database. Networking is interesting because it’s an area that is running counter-Moore and actually getting relatively more expensive while the rest of the industry is going the other way. And, at the same time that is happening, network traffic is increasing at rates faster than server deployments. We’ll lay out how we have fundamentally reengineered our network for more capacity, lower latency, and reduced jitter while lowering costs. Database is my second area of focus. It’s another area where we feel substantial new engineering investment can give developers better solutions at far lower costs partly based upon open source, partly based upon shifting the bulk of the complexity of database administration to AWS, and partly based upon new engine development optimized for cloud deployments. There is lots to cover in both networking and database but I’ll also show how AWS processes 10s of millions of records per second internally and how this solution has become an externally available cloud service.


I’ll post the slides I present once available to:

Live streaming for select sessions available at:


--James Hamilton,


Friday, November 07, 2014 9:36:06 PM (Pacific Standard Time, UTC-08:00)  #    Comments [5] - Trackback
 Wednesday, July 02, 2014

We all know that when designing and operating applications at scale, it is persistent state management that brings the most difficult challenges. Delivering state-free applications has always been (fairly) easy. But most interesting commercial and consumer applications need to manage persistent state. Advertising needs to be delivered, customer activity needs to be tracked, and products need to be purchased. Interesting applications that run at scale all have difficult persistent state problems. That’s why, other AWS customers, and even other AWS services make use of the various AWS database platform services. Delegating the challenge of managing high-performance, transactional, and distributed data management to underlying services makes applications more robust while reducing operational overhead and design complexity. But, it makes these mega-scale database systems absolutely critical resources that need to be rock solid with very well understood scaling capabilities.

One such service that is heavily used by AWS,, and external services is DynamoDB. On the surface, DynamoDB looks fairly simple. It has only 13 APIs and has been a service that is being widely adopted due to its ease of use and doing the undifferentiated heavy lifting of dealing with disk failures, host issues or network partitions etc. However, that external simplicity is built on a hidden substrate of complex distributed systems. Such complex internals are required to achieve high-availability while running on cost-efficient infrastructure, and also to cope with rapid business-growth. As an example of this growth, DynamoDB is handling millions of transactions every second in a single region while continuing to provide single digit millisecond latency even in the midst of disk failures, host failures and rack power disruptions.

Services like DynamoDB rely on fault-tolerant distributed algorithms for replication, consistency, concurrency control, provisioning, and auto scaling. There are many such algorithms in the literature, but combining them into a cohesive system is a significant challenge, as the algorithms usually need to be modified in order to interact properly in a real-world production system. In addition, we have found it necessary to invent algorithms of our own.

We all know that increasing complexity greatly magnifies the probability of human error in design, code, and operations. Errors in the core of the system could potentially cause service disruptions and impact our service availability goals. We work hard to avoid unnecessary complexity, but the complexity of the task remains high. Before launching a service or a new feature to service like DynamoDB, we need to reach extremely high confidence that the core of the system is correct.

Historically, software companies do this using a combination of design documents, design reviews, static code analysis, code reviews, stress testing, fault injection testing and many other techniques. While these techniques are necessary they may not be sufficient. If you are building a highly concurrent replication algorithm that serves as the backbone for systems like DynamoDB, you want to be able to model partial failures in a highly concurrent distributed systems. Moreover, you want to capture the failures at design level even as these might be harder to do in testing.

This is where AWS teams and Amazon DynamoDB and transactional services teams embarked on a path of building precise designs for the services they are building.  While one can argue that traditional methods of writing design docs serve a similar purpose we found that design docs lack precision. This is because they are written in prose and diagrams and they are not easy to test in an automated fashion. At the other end of the spectrum, once we have implemented the system in code, it becomes much too complex to establish algorithm correctness or to debug subtle issues. To this end, we looked for a way to express designs that will eliminate hand waving and that has sufficient tools that can be applied to check for errors in the design.

We wanted a language that allowed us to express things like replication algorithms or distributed algorithms in hundreds of lines of code. We wanted the tool to have existing ecosystems that allowed us to test various failure conditions at design level quickly. We found what we were looking for in TLA+, a formal specification language invented by ACM Turing award winner, Leslie Lamport. TLA+ is based on simple discrete math, basic set theory and predicates with which all engineers are quite familiar. A TLA+ specification simply describes the set of all possible legal behaviors (execution traces) of a system. While TLA+ syntax was unfamiliar to our engineers, TLA+ is accompanied by PlusCal, a pseudo code language that is closer to a C-style programming language. Several engineers at Amazon have found they are more productive in PlusCal than TLA+. However, in other cases, the additional flexibility of plain TLA+ has been very useful. 

PlusCal and TLA+ have proven very effective at establishing and maintaining the correctness through change of the fundamental components on which DynamoDB is based. We believe having these core components correct from day one has allowed the DynamoDB system to evolve more quickly and scale fast while avoiding the difficult times often experienced by engineers and customers early in a distributed systems life.

I’ve always been somewhat skeptical of formal methods in that some bring too much complexity to actually be applied to commercial systems while others tend to abstract much of the complexity but, in abstracting away complexity, give up precision. TLA+ and PlusCal appear to skirt these limitations and we believe that having the core and most fundamental algorithms on which DynamoDB is based provably correct helps speed innovation, ease the inherent complexity of scaling and, overall, improves both the customer experience and the experience of engineers working on the DynamoDB system at AWS. While this is an important part of our system, there are hundreds of innovations we do in building and operating robust scalable distributed systems on how we design, build, test, deploy, monitor and operate these large scale services.

If you are interested in reading more about the usage of TLA+ within AWS, this is a pretty good source: Use of Formal Methods at Amazon Web Services. And, if you are interested in joining the AWS database team and, especially if you have the experience, background, and interest to lead a team in our database group, drop me a note with your current resume ( We’re expanding into new database areas quickly and growing existing services even faster.

James Hamilton,


Wednesday, July 02, 2014 4:04:52 PM (Pacific Standard Time, UTC-08:00)  #    Comments [11] - Trackback
 Friday, February 14, 2014

Most agree that cloud computing is inherently more efficient that on premise computing in each of several dimensions. Last November, I went after two of the easiest to argue gains: utilization and the ability to sell excess capacity (Datacenter Renewable Power Done Right):


Cloud computing is a fundamentally more efficiently way to operate compute infrastructure. The increases in efficiency driven by the cloud are many but a strong primary driver is increased utilization. All companies have to provision their compute infrastructure for peak usage. But, they only monetize the actual usage which goes up and down over time. What this leads to incredibly low average utilization levels with 30% being extraordinarily high and 10 to 20% the norm.  Cloud computing gets an easy win on this dimension. When non-correlated workloads from a diverse set of customers are hosted in the cloud, the peak to average flattens dramatically.  Immediately effective utilization sky rockets.  Where 70 to 80% of the resources are usually wasted, this number climbs rapidly with scale in cloud computing services flattening the peak to average ratio.


To further increase the efficiency of the cloud, Amazon Web Services added an interesting innovation where they sell the remaining capacity not fully consumed by this natural flattening of the peak to average. These troughs are sold on a spot market and customers are often able to buy computing at less the amortized cost of the equipment they are using (Amazon EC2 Spot Instances). Customers get clear benefit. And, it turns out, it’s profitable to sell unused capacity at any price over the marginal cost of power. This means the provider gets clear benefit as well. And, with higher utilization, the environment gets clear benefit as well.


Back in June, Lawrence Berkeley National Labs released a study that went after the same question quantitatively and across a much broader set of dimensions.  I first came across the report via coverage in Network Computing: Cloud Data Centers: Power Savings or Power drain? The paper was funded by Google which admittedly has an interest in cloud computing and high scale computing in general. But, even understanding that possible bias or influence, the paper is of interest. From Google’s summary of the findings (How Green is the Internet?):


Funded by Google, Lawrence Berkeley National Laboratory investigated the energy impact of cloud computing. Their research indicates that moving all office workers in the United States to the cloud could reduce the energy used by information technology by up to 87%.


These energy savings are mainly driven by increased data center efficiency when using cloud services (email, calendars, and more). The cloud supports many products at a time, so it can more efficiently distribute resources among many users. That means we can do more with less energy.


The paper attempts to quantify the gains achieved by moving workloads to the cloud by looking at all relevant dimensions of savings some fairly small and some quite substantial. From my perspective, there is room to debate any of the data one way or the other but the case is sufficiently clear that it’s hard to argue that there aren’t substantial environmental gains.  


Now, what I would really love to see is an analysis of an inefficient, poorly utilized private datacenter who’s operators wants to be green and needs to compare the installation of a fossil fuel consuming fuel cell power system with dropping the same workload down onto one of the major cloud computing platforms :-).  


The paper is in full available at: The Energy Efficiency Potential of Cloud-Based Software: A US Case Study.


James Hamilton 
b: /


Friday, February 14, 2014 6:37:17 PM (Pacific Standard Time, UTC-08:00)  #    Comments [2] - Trackback
 Saturday, January 04, 2014

It’s not often I’m enthused about spending time in Las Vegas but this year’s AWS re:Invent conference was a good reason to be there. It’s exciting getting a chance to meet with customers who have committed their business to the cloud or are wrestling with that decision.


The pace of growth since last years was startling but what really caught my attention was the number of companies that had made the transition between testing on the cloud to committing their most valuable workloads to run there. I fully expected this to happen but I’ve seen these industry sweeping transitions before. They normally take a long, long time. This pace of the transition to the cloud is, at least in my experience, unprecedented.


I did a 15 min video interview with SiliconANGLE on Wednesday that has been poste:

·         Video:


I did a talk on Thursday that is posted as well:

·         Slides:

·         Video:


The talk focused mainly on the question of is the cloud really different?  Can’t I just get a virtualization platform and do a private cloud and enjoy the same advantages? To investigate that question, I started by looking at AWS scale since scale is the foundation on which all the other innovations are built. It’s scale that funds the innovations and it’s scale that makes even small gains worth pursuing since the multiplier is so large. I then  walked through some of the innovations happening in AWS and talked about why they probably wouldn’t make sense for an on-premise deployment.  Looking at the innovations I tried to show the breadth of what was possible by sampling from many different domains including custom high voltage power substations, custom server design, custom networking hardware, a networking protocol development team, purpose built storage rack designs, the AWS Spot market, and supply chain and procurement optimization.


On scale, I laid out several data points that we hadn’t been talking about externally in the past but still used my favorite that is worth repeating here: Every day AWS adds enough new server capacity to support all of Amazon’s global infrastructure when it was a $7B annual enterprise. This point more than any other underlines what I was saying earlier: big companies are moving to the cloud and this transition is happening faster than any industry transition I’ve ever seen.


Each day enough servers are deployed to support a $7B ecommerce company. It happens every day and the pace continues to accelerate. Just imagine the complexity of getting all that gear installed each day. Forget innovation. Just the logistical challenge is a pretty interesting problem.


The cloud is happening in a big way, the pace is somewhere between exciting and staggering, and any company that isn’t running at least some workloads in the cloud and learning is making a mistake.


If you have time to attend re:Invent next year, give it some thought. More than 1/3 of the sessions are from customers talking about what they have deployed and how they did it and it’s an interesting and educational week. Most of the re:Invent talks are posted at:


I’ve not been doing a good job of posting talks to as I should so here’s a cummulative update:

·         2013.11.14: Why Scale Matters and How the Cloud is Different (talk, video), re:Invent 2013, Las Vegas, NV

·         2013.11.13: AWS The Cube at AWS re:Invent 2013 with James Hamilton (video), re:Invent 2013, Las Vegas NV

·         2013.01.22: Infrastructure Innovation Opportunities (talk), Ycombinator, Mountain View, CA

·         2012.11.29: Failures at Scale and how to Ride Through Them (talk, video), re:Invent 2012, Las Vegas, NV

·         2012.11.28: Building Web Scale Applications with AWS (talk, video), re:Invent 2012, Las Vegas, NV

·         2012.10.09: Infrastructure Innovation Opportunities (talk, video), First Round Capital CTO Summit, San Francisco, CA.

·         2012.09.25: Cloud Computing Driving Infrastructure Innovation (talk), Intel Distinguished Speaker Series, Hillsboro, OR


James Hamilton 
b: /


Saturday, January 04, 2014 10:53:08 AM (Pacific Standard Time, UTC-08:00)  #    Comments [3] - Trackback
 Saturday, November 30, 2013

Facebook Iowa Data Center

In 2007, the EPA released a study on datacenter power consumption at the request of the US Congress (EPA Report to Congress on Server and Data Center Efficiency).  The report estimated that the power consumption of datacenters represented about 1.5% of the US Energy Budget in 2005 and this number would double by 2010. In a way, this report was believable in that datacenter usage was clearly on the increase. What the report didn’t predict was the pace of innovation in datacenter efficiency during that period.  Increased use, spurred increased investment, which has led to a near 50% improvement in industry leading datacenter efficiency. 


Also difficult to predict at the time of the report was the rapid growth of cloud computing. Cloud computing is a fundamentally more efficiently way to operate compute infrastructure. The increases in efficiency driven by the cloud are many but a strong primary driver is increased utilization. All companies have to provision their compute infrastructure for peak usage. But, they only monetize the actual usage which goes up and down over time. What this leads to incredibly low average utilization levels with 30% being extraordinarily high and 10 to 20% the norm.  Cloud computing gets an easy win on this dimension. When non-correlated workloads from a diverse set of customers are hosted in the cloud, the peak to average flattens dramatically.  Immediately effective utilization sky rockets.  Where 70 to 80% of the resources are usually wasted, this number climbs rapidly with scale in cloud computing services flattening the peak to average ratio.


To further increase the efficiency of the cloud, Amazon Web Services added an interesting innovation where they sell the remaining capacity not fully consumed by this natural flattening of the peak to average. These troughs are sold on a spot market and customers are often able to buy computing at less the amortized cost of the equipment they are using (Amazon EC2 Spot Instances). Customers get clear benefit. And, it turns out, it’s profitable to sell unused capacity at any price over the marginal cost of power. This means the provider gets clear benefit as well. And, with higher utilization, the environment gets clear benefit as well.


It’s easy to see why the EPA datacenter power predictions were incorrect and, in fact, the most recent data shows just how far off the original estimates actually were. The EPA report estimated that in 2005, the datacenter industry was consuming just 1.5% of the nation’s energy budget but what really caused international concern was the projection that this percentage would surge to 3.0% in 2010. This 3% in 2010 number has been repeated so many times that it’s now considered a fact by many and it has to have been one of the most referenced statistics in the datacenter world.


The data on 2010 is now available and the power consumption of datacenters in 2010 is currently estimated to have been in the 1.1 to 1.5% in 2010. Almost no changes since 2005. Usage has skyrocketed. Most photos are ending up in the cloud, social networking is incredibly heavily used, computational science has exploded, machine learning is tackling business problems previously not cost effective to address, much of the world now have portable devices and a rapidly increasing percentage of these devices are cloud connected. Server side computing is exploding and yet power consumption is not.


However, the report caused real concern both in the popular press and across the industry and there have been many efforts to not only increase efficiency but to also to increase the usage of clean power. The progress on efficiency has been stellar but the early efforts on green power have been less exciting.


I’ve mentioned some of the efforts to go after cleaner data center power in the past. The solar array at the Facebook Prineville datacenter is a good example: It’s a 100kW solar array “powering” a 25MW+ facility. Pure optics with little contribution to the datacenter power mix (I love solar but… ). Higher scale efforts such as the Apple Maiden North Carolina facility are getting much closer to being able to power the entire facility go far beyond mere marketing programs. But, for me, the land consumed is disappointing with 171 acres of land cleared of trees in order to provide green power. There is no question that solar can augment the power at major datacenters but onsite generation with solar just doesn’t look practical for most facilities particularly those in densely packed urban settings. In the article Solar at Scale: How Big is a Solar Array with 9MW Average Output, we see that a 9MW average output solar plant takes approximately 314 acres (13.6 million sq/ft). This power density is not practical for onsite generation at most datacenters and may not be the best approach to clean power for datacenters.


The biggest issue with solar applied to datacenters is the mass space requirements are not practical with most datacenters being located in urban centers and it may not even be the right choice for rural centers. Another approach gaining rapid popularity is the use of fuel cells to power datacenters. At least partly because of highly skilled marketing, fuel cells are considered by many jurisdictions to be “green”. The good news is that some actually are green. A fuel cell running on waste biogas is indeed a green solution with considerable upside. This is what Apple plans for its Maiden North Carolina fuel cell farm and it’s a very nice solution (Apple’s Biogas Fuel Cell Plant Could Go Live by June).


The challenge with fuel cells in the datacenter application is just about all of them are not biogas powered relying instead on non-renewable energy. For example, the widely heralded eBay Salt Lake City fuel cell deployment is natural gas powered (New eBay Data Center Runs Almost Entirely on Bloom Fuel Cells). There have been many criticisms of the efficiency of these fuel cell-based power plants (884 may be Bloom Energy’s Fatal Number and their cost effectiveness (Delaware Press Wakes Up to Bloom Energy Boondoggle).


I’m mostly ignoring these reports and focusing on the simpler question: how can running a low-scale power plant on a fossil fuel possibly be green and is on-site generation really going to be the most environmentally conscious means of powering data centers? Most datacenters lack space for large on-site generation facilities, most datacenter operators don’t have the skill, and generally the most efficient power generation plants are far bigger than a single datacenter could ever consume on its worst day.


The argument for on-site generation is to avoid the power lost in transmission. The US Energy Information Administration (EIA)  estimates power lost in transmission to be 7%.  That’s certainly a non-zero number and the loss is relevant but I still find myself skeptical that the world leaders in operating efficient and reliable power plans are going to be datacenter operators. And, even if that prospect does excite you, does having a single plant on one location being the sole power source for the datacenter sounds like a good solution? I’m finding it hard to get excited. Maintenance, natural disasters, and human error will eventually yield some unintended consequences.


A solution that I really like is the Facebook Altoona Iowa facility (Facebook Says Its New Data Center Will Run Entirely on Wind). It is a standard grid connected datacenter without on-site power generation. But they have partnered to have built a 138MW wind project in nearby Wellsburg Iowa. What I like about this approach is: 1) no clear cutting was required to prepare the land for generation and the land remains multi-use, 2) it’s not fossil fuel powered, 3) the facility will be run by a major power generation operator rather than as a sideline by the datacenter operator, and 4) far more clean power is being produced than will be actually used by the datacenter so they are actually adding more clean power to the grid than they are consuming by a fairly significant margin.


Hats off to Facebook for doing it clean data center energy right. Nice work.


James Hamilton 
b: /


Saturday, November 30, 2013 1:38:31 PM (Pacific Standard Time, UTC-08:00)  #    Comments [6] - Trackback
 Tuesday, July 16, 2013

At the Microsoft World-Wide Partners Conference, Microsoft CEO Steve Ballmer announced that “We have something over a million servers in our data center infrastructure. Google is bigger than we are. Amazon is a little bit smaller. You get Yahoo! and Facebook, and then everybody else is 100,000 units probably or less.


That’s a surprising data point for a variety of reasons. The most surprising is that the data point was released at all. Just about nobody at the top of the server world chooses to boast with the server count data point. Partly because it’s not all that useful a number but mostly because a single data point is open to a lot of misinterpretation by even skilled industry observers. Basically, it’s pretty hard to see the value of talking about server counts and it is very easy to see the many negative implications that follow from such a number


The first question when thinking about this number is where does the comparative data actually come from?  I know for sure that Amazon has never released server count data. Google hasn’t either although estimates of their server footprint abound. Interestingly the estimates of Google server counts 5 years ago was 1,000,000 servers whereas current estimates have them only in the 900k to 1m range.


The Microsoft number is surprising when compared against past external estimates, data center build rates, or ramp rates from previous hints and estimates. But, as I said, little data has been released by any of the large players and what’s out there is typically nothing more than speculation. Counting servers is hard and credibly comparing server counts is close to impossible.


Assume that each server runs 150 to 300W including all server components with a weighted average of say 250W/server. And as a Power Usage Effectiveness estimator, we will use 1.2 (only 16.7% of the power is lost to datacenter cooling and power distribution losses). With these scaling points, the implied total power consumption is over 300MW. Three hundred million watts or, looking at annual MW-h, we get an annual consumption of over 2,629,743 MWh or 2.6 terawatt hours – that’s a hefty slice of power even by my standards. As a point of scale since these are big numbers, the US Energy Information Administration reports that in 2011 the average US household consumed 11.28MWh. Using that data point, 2.6TWh is just a bit more than the power consumed by 230,000 US homes.


Continuing through the data and thinking through what follows from “over 1M” servers, the capital expense of servers will be $1.45 billion dollars assuming a very inexpensive $1,450 per server. Assuming a mix of different servers with an average cost of $2,000/server, the overall capital expense would be $2B before looking at the data center infrastructure and networking costs required to house them. With the overall power consumption computed above of 300MW which is 250MW of critical power (using a PUE of 1.2) and assuming a data center build cost at a low $9/W of critical load (Uptime Institute estimates numbers closer to $12/W), we would have a data center build cost of $2.25B dollars.  The implication of “over 1M servers” is an infrastructure investment of $4.25B including the servers. That’s an athletic expense even for Microsoft but certainly possible.


How many datacenters would be implied by “more than one million servers?” Ignoring the small points of presence since they don’t move the needle, and focusing on the big centers, let’s assume 50,000 servers in each facility. That assumption would lead to 30 major facilities.  As a cross check, if we instead focus on power consumption as a way to compute facility count and assume a total datacenter power consumption of 20MW each and the previously computed 300MW total power consumption, we would have roughly 15 large facilities. Not an unreasonable number in this context.


The summary following from these data and the “over 1M servers” number:

·         Facilities: 15 to 30 large facilities

·         Capital expense: $4.25B

·         Total power: 300MW

·         Power Consumption: 2.6TWh annually


Over one million servers is a pretty big number even in a web scaled world.




Data Center Knowledge article: Ballmer: Microsoft has 1 Million Servers

Transcript of Ballmer presentation:  Steve Ballmer: World-Wide Partner Conference Keynote

Additional note: Todd Hoff of High Scalability made a fun observation in his post Steve Ballmer says Microsoft has over 1 million servers. From Todd: “The [Microsoft data center] power consumption is about the same as used by Nicaragua and the capital cost is about a third of what Americans spent on video games in 2012.

 James Hamilton 

b: /



Tuesday, July 16, 2013 4:34:17 PM (Pacific Standard Time, UTC-08:00)  #    Comments [11] - Trackback
 Monday, January 14, 2013

In the cloud there is nothing more important than customer trust. Without customer trust, a cloud business can’t succeed. When you are taking care of someone else’s assets, you have to treat those assets as more important than your own. Security has to be rock solid and absolutely unassailable. Data loss or data corruption has to be close to impossible and incredibly rare.  And all commitments to customers have to be respected through business changes. These are hard standards to meet but, without success against these standards, a cloud service will always fail. Customers can leave any time and, if they have to leave, they will remember you did this to them.


These are facts and anyone working in cloud services labors under these requirements every day. It’s almost reflexive and nearly second nature. What brought this up for me over the weekend was a note I got from one of my cloud service providers. It emphasized that it really is worth talking more about customer trust.


Let’s start with some history.  Many years ago, Michael Merhej and Tom Klienpeter started a company called ByteTaxi that eventually offered a product called Foldershare. It was a simple service with a simple UI but it did peer-to-peer file sync incredibly well, it did it through firewalls, it did it without install confusion and, well, it just worked. It was a simple service but was well executed and very useful. In 2005, Microsoft acquired Foldershare and continued to offer the service. It didn’t get enhanced much for years but it remained useful.  Then Microsoft came up with a broader plan called Windows Live Mesh and the Foldershare service was renamed. Actually the core peer-to-peer functionality passed through an array of names and implementations from Foldershare, Windows Live Foldershare, Windows Live Sync and finally Windows Live Mesh.


During the early days at Microsoft, it was virtually uncared for and had little developer attention. As new names and implementations were announced and the feature actually had developer attention, it was getting enhanced but, ironically, it was also getting somewhat harder to use and definitely less stable. But, it still worked and the functionality lived on in Live Mesh. Microsoft has another service called Skydrive that does the same thing that all the other cloud sync services do: sync files to cloud hosted storage. Unfortunately, it doesn’t include the core peer-to-peer functionality of Live Mesh.  Reportedly 40% of the Live Mesh users also use Skydrive.


This is where we get back to customer trust. Over the weekend, Microsoft sent out a note to all Mesh users confirming it will be shut off next month as a follow up to their announcement that the service will be killed that went out in December. They explained the reason to terminate the service and remove the peer-to-peer file sync functionality:


Currently 40% of Mesh customers are actively using SkyDrive and based on the positive response and our increasing focus on improving personal cloud storage, it makes sense to merge SkyDrive and Mesh into a single product for anytime and anywhere access for files.


Live Mesh is being killed without a replacement service.  It’s not a big deal but 2 months isn’t a lot of warning. I know that this sort of thing can happen to small startups anytime and, at any time, customers could get left unsupported. But, Microsoft seems well beyond the startup phase at this point.  I get that strategic decisions have to be made but there are times when I wonder how much thought went into the decision. I suspect it was something like “there are only 3 million Live Mesh customers so it’s really not worth continuing with it.” And, it actually may not be worth continuing the service. But, there is this customer trust thing. And I just hate to see it violated – it’s bad for all cloud provider when anyone in the industry makes a decision that raises the customer trust question.


Fortunately, there is a Mesh replacement service:  I’ve been using it since the early days when it was in controlled beta. Over the last month or so Cubby has moved to full, unrestricted production. It’s been solid for the period I’ve been using it and, like Foldershare, its simple and it works. I really like it. If you are a Mesh user, were a Foldershare user, or just would like to be able to sync your files between your different systems, try Cubby.  Cubby also add support for Android or IOS devices without extra cost. Cubby is well executed and stable.


It must be Cloud Cleaning week at Microsoft.  A friend forwarded the note sent to the millions of active Microsoft Messenger customers this month: the service is being “retired” and users are recommended to consider Skype.


If you are interested in reading more on the Live Mesh service elimination, the following is the text of the note sent to all current Mesh users:

Dear Mesh customer,

Recently we released the latest version of SkyDrive, which you can use to:

  • Choose the files and folders on your SkyDrive that sync on each computer.
  • Access your SkyDrive using a brand new app for Android v2.3 or the updated apps for Windows Phone, iPhone, and iPad.
  • Collaborate online with the new Office Web apps, including Excel forms, co-authoring in PowerPoint and embeddable Word documents.

Currently 40% of Mesh customers are actively using SkyDrive and based on the positive response and our increasing focus on improving personal cloud storage, it makes sense to merge SkyDrive and Mesh into a single product for anytime and anywhere access for files. As a result, we will retire Mesh on February 13, 2013. After this date, some Mesh functions, such as remote desktop and peer to peer sync, will no longer be available and any data on the Mesh cloud, called Mesh synced storage or SkyDrive synced storage, will be removed. The folders you synced with Mesh will stop syncing, and you will not be able to connect to your PCs remotely using Mesh.

We encourage you to try out the new SkyDrive to see how it can meet your needs. During the transition period, we suggest that, in addition to using Mesh, you sync your Mesh files using SkyDrive. This way, you can try out SkyDrive without changing your existing Mesh setup. For tips on transitioning to SkyDrive, see SkyDrive for Mesh users on the Windows website. If you have questions, you can post them in the SkyDrive forums.

Mesh customers have been influential and your feedback has helped shape our strategy for Mesh and SkyDrive. We would not be here without your support and hope you continue to give us feedback as you use SkyDrive.


The Windows Live Mesh and SkyDrive teams


There is real danger of thinking of customers as faceless aggregations of hundreds of thousands or even millions of users. We need to think through decisions one user at a time and make it work for them individually. If millions of active users are on Microsoft Messenger, what would it take to make them want to use Skype?  If 60% of the Windows Live Mesh users chose not to use Microsoft Skydrive, why is that? Considering customers one at a time is clearly the right thing for customers but, long haul, it’s also the right thing for the business. It builds the most important asset in the cloud, customer trust.




James Hamilton 
b: /

Monday, January 14, 2013 8:32:32 PM (Pacific Standard Time, UTC-08:00)  #    Comments [16] - Trackback
 Wednesday, November 28, 2012

I’ve worked in or near the database engine world for more than 25 years. And, ironically, every company I’ve ever worked at has been working on a massive-scale, parallel, clustered RDBMS system. The earliest variant was IBM DB2 Parallel Edition released in the mid-90s. It’s now called the Database Partitioning Feature.


Massive, multi-node parallelism is the only way to scale a relational database system so these systems can be incredibly important. Very high-scale MapReduce systems are an excellent alternative for  many workloads. But some customers and workloads want the flexibility and power of being able to run ad hoc SQL queries against petabyte sized databases. These are the workloads targeted by massive, multi-node relational database clusters and there are now many solutions out there with Oracle RAC being perhaps the most well-known but there are many others including Vertica, GreenPlum, Aster Data, ParAccel, Netezza, and Teradata.


What’s common across all these products is that big databases are very expensive. Today, that is changing with the release of Amazon Redshift. It’s a relational, column-oriented, compressed, shared nothing, fully managed, cloud hosted, data warehouse. Each node can store up to 16TB of compressed data and up to 100 nodes are supported in a single cluster.


Amazon Redshift manages all the work needed to set up, operate, and scale a data warehouse cluster, from provisioning capacity to monitoring and backing up the cluster, to applying patches and upgrades. Scaling a cluster to improve performance or increase capacity is simple and incurs no downtime. The service continuously monitors the health of the cluster and automatically replaces any component, if needed.


The core node on which the Redshift clusters are build, includes 24 disk drives with an aggregate capacity of 16TB of local storage. Each node has 16 virtual cores and 120 Gig of memory and is connected via a high speed 10Gbps, non-blocking network. This a meaty core node and Redshift supports up to 100 of these in a single cluster.


There are many pricing options available (see for more detail) but the most favorable comes in at only $999 per TB per year. I find it amazing to think of having the services of an enterprise scale data warehouse for under a thousand dollars by terabyte per year. And, this is a fully managed system so much of the administrative load is take care of by Amazon Web Services.


Service highlights from:


Fast and Powerful – Amazon Redshift uses a variety to innovations to obtain very high query performance on datasets ranging in size from hundreds of gigabytes to a petabyte or more. First, it uses columnar storage and data compression to reduce the amount of IO needed to perform queries. Second, it runs on hardware that is optimized for data warehousing, with local attached storage and 10GigE network connections between nodes. Finally, it has a massively parallel processing (MPP) architecture, which enables you to scale up or down, without downtime, as your performance and storage needs change.

You have a choice of two node types when provisioning your own cluster, an extra large node (XL) with 2TB of compressed storage or an eight extra large node (8XL) with 16TB of compressed storage. You can start with a single XL node and scale up to a 100 node eight extra large cluster. XL clusters can contain 1 to 32 nodes while 8XL clusters can contain 2 to 100 nodes.


Scalable – With a few clicks of the AWS Management Console or a simple API call, you can easily scale the number of nodes in your data warehouse to improve performance or increase capacity, without incurring downtime. Amazon Redshift enables you to start with a single 2TB XL node and scale up to a hundred 16TB 8XL nodes for 1.6PB of compressed user data. Resize functionality is not available during the limited preview but will be available when the service launches.


Inexpensive – You pay very low rates and only for the resources you actually provision. You benefit from the option of On-Demand pricing with no up-front or long-term commitments, or even lower rates via our reserved pricing option. On-demand pricing starts at just $0.85 per hour for a two terabyte data warehouse, scaling linearly up to a petabyte and more. Reserved Instance pricing lowers the effective price to $0.228 per hour, under $1,000 per terabyte per year.

Fully Managed – Amazon Redshift manages all the work needed to set up, operate, and scale a data warehouse, from provisioning capacity to monitoring and backing up the cluster, and to applying patches and upgrades. By handling all these time consuming, labor-intensive tasks, Amazon Redshift frees you up to focus on your data and business insights.


Secure – Amazon Redshift provides a number of mechanisms to secure your data warehouse cluster. It currently supports SSL to encrypt data in transit, includes web service interfaces to configure firewall settings that control network access to your data warehouse, and enables you to create users within your data warehouse cluster. When the service launches, we plan to support encrypting data at rest and Amazon Virtual Private Cloud (Amazon VPC).


Reliable – Amazon Redshift has multiple features that enhance the reliability of your data warehouse cluster. All data written to a node in your cluster is automatically replicated to other nodes within the cluster and all data is continuously backed up to Amazon S3. Amazon Redshift continuously monitors the health of the cluster and automatically replaces any component, as necessary.


Compatible – Amazon Redshift is certified by Jaspersoft and Microstrategy, with additional business intelligence tools coming soon. You can connect your SQL client or business intelligence tool to your Amazon Redshift data warehouse cluster using standard PostgreSQL JBDBC or ODBC drivers.


Designed for use with other AWS Services – Amazon Redshift is integrated with other AWS services and has built in commands to load data in parallel to each node from Amazon Simple Storage Service (S3) and Amazon DynamoDB, with support for Amazon Relational Database Service and Amazon Elastic MapReduce coming soon.


Petabyte-scale data warehouses no longer need command retail prices of upwards $80,000 per core. You don’t have to negotiate an enterprise deal and work hard to get the 60 to 80% discount that always seems magically possible in the enterprise software world.  You don’t even have to hire a team of administrators. Just load the data and get going. Nice to see.




James Hamilton 
b: /


Wednesday, November 28, 2012 9:37:51 AM (Pacific Standard Time, UTC-08:00)  #    Comments [6] - Trackback
 Thursday, October 11, 2012

The last few weeks have been busy and it has been way too long since I have blogged. I’m currently thinking through the server tax and what’s wrong with the current server hardware ecosystem but don’t have anything yet ready to go on that just yet. But, there are a few other things on the go.  I did a talk at Intel a couple of weeks back and last week at the First Round Capital CTO summit. I’ve summarized what I covered below with pointers to slides. 


In addition, I’ll be at the Amazon in Palo Alto event this evening and will do a talk there as well. If you are interested in Amazon in general or in AWS specifically, we have a new office open in Palo Alto and you are welcome to come down this evening to learn more about AWS, have a beer or the refreshment of your choice, and talk about scalable systems. Feel free to attend if you are interested:


Amazon in Palo Alto

October 11, 2012 at 5:00 PM - 9:00 PM

Pampas 529 Alma Street Palo Alto, CA 94301



First Round Capital CTO Summit:



I started this session by arguing that cost or value models are the right way to ensure you are working on the right problem. I come across far too many engineers and even companies that are working on interesting problems but they fail at the “top 10 problem” test. You never want to first have to explain the problem to a perspective customer before you get a chance to explain your solution. It is way more rewarding to be working on top 10 problems where the value of what you are doing is obvious and you only need to convince someone that your solution actually works.


Cost models are a good way to force yourself to really understand all aspects of what the customer is doing and know precisely what savings or advantage you bring. A 25% improvement on an 80% problem is way better than  50% solution to a 5% problem. Cost or value models are a great way of keeping yourself honest on what the real savings or improvement of your approach actually are. And its quantifiable data that you can verify in early tests and prove in alpha or beta deployments.


I then covered three areas of infrastructure where I see considerable innovation and showed all the cost model helped drive me there:


·         Networking: The networking eco-system is still operating on the closed, vertically integrated, mainframe model but the ingredients are now in place to change this. See Networking, the Last Bastion of Mainframe Computing for more detail. The industry is currently going through great change. Big change is a hard transition for the established high-margin industry players but it’s a huge opportunity for startups.

·         Storage: The storage (and database) worlds are going through a unprecedented change where all high-performance random access storage is migrating from hard disk drives to flash storage. The early flash storage players have focused on performance over price so there is still considerable room for innovation. Another change happening in the industry is the explosion of cold storage (low I/O density storage that I jokingly refer to as write-only) due to falling prices, increasing compliance requirements, and an industry realization that data has great value. This explosion in cold storage is opening much innovation and many startup opportunities.  The AWS entrant in this market is Glacier where you can store seldom accessed data at one penny per GB per month (for more on Glacier: Glacier: Engineering for Cold Storage in the Cloud.

·         Cloud Computing: I used to argue that targeting cloud computing was a terrible idea for startups since the biggest cloud operators like Google and Amazon tend to do all custom hardware and software and purchase very little commercially. I may have been correct initially but, with the cloud market growing so incredibly fast, every teleco is entering the market, each colo provider is entering, most hardware providers are entering, … the number of players is going from 10s to 1000s. And, at 1,000s, it’s a great market for a startup to target. Most of these companies are not going to build custom networking, server, and storage hardware but they do have the need to innovate with the rest of the industry.


Slides: First Round Capital CTO Summit


Intel Distinguished Speaker Series:


In this talk I started with how fast the cloud computing market segment is growing using examples form AWS. I then talked about why cloud computing is such an incredible customer value proposition. This isn’t just a short term fad that will pass over time. I mostly focused on how that statement I occasionally hear just can’t be possibly be correct: “I can run my on-premise computing infrastructure less expensively then hosting it in the cloud”. I walk through some of the reasons why this statement can only be made with partial knowledge. There are reasons why some computing will be in the cloud and some will be hosted locally and industry transitions absolutely do take time but cost isn’t one of the reasons that some workloads aren’t in the cloud.


I think walked through 5 areas of infrastructure innovation and some of what is happening in each area:

·         Power Distribution

·         Mechanical Systems

·         Data Center Building Design

·         Networking

·         Storage


Slides: Intel Distinguished Speaker Series


I hope to see you tonight at the Amazon Palo Alto event at Pampas ( The event starts at 5pm and I’ll do a short talk at 6:35.



Thursday, October 11, 2012 10:04:49 AM (Pacific Standard Time, UTC-08:00)  #    Comments [0] - Trackback
Ramblings | Services
 Tuesday, August 21, 2012

Earlier today Amazon Web Services announced Glacier, a low-cost, cloud-hosted, cold storage solution. Cold storage is a class of storage that is discussed infrequently and yet it is by far the largest storage class of them all. Ironically, the storage we usually talk about and the storage I’ve worked on for most of my life is the high-IOPS rate storage supporting mission critical databases. These systems today are best hosted on NAND flash and I’ve been talking recently about two AWS solutions to address this storage class:



Cold storage is different. It’s the only product I’ve ever worked upon where the customer requirements are single dimensional. With most products, the solution space is complex and, even when some customers may like a competitive product better for some applications, your product still may win in another. Cold storage is pure and unidimensional.  There is only really one metric of interest: cost per capacity. It’s an undifferentiated requirement that the data be secure and very highly durable. These are essentially table stakes in that no solution is worth considering if it’s not rock solid on durability and security.  But, the only dimension of differentiation is price/GB.


Cold storage is unusual because the focus needs to be singular. How can we deliver the best price per capacity now and continue to reduce it over time? The focus on price over performance, price over latency, price over bandwidth actually made the problem more interesting. With most products and services, it’s usually possible to be the best on at least some dimensions even if not on all. On cold storage, to be successful, the price per capacity target needs to be hit.  On Glacier, the entire project was focused on delivering $0.01/GB/Month with high redundancy and security and to be on a technology base where the price can keep coming down over time. Cold storage is elegant in its simplicity and, although the margins will be slim, the volume of cold storage data in the world is stupendous. It’s a very large market segment. All storage in all tiers backs up to the cold storage tier so its provably bigger than all the rest. Audit logs end up in cold storage as do web logs, security logs, seldom accessed compliance data, and all other data I refer jokingly to as Write Only Storage. It turns out that most files in active storage tiers are actually never accessed (Measurement and Analysis of Large Scale Network File System Workloads ). In cold storage, this trend is even more extreme where reading a storage object is the exception. But, the objects absolutely have to be there when needed. Backups aren’t needed often and compliance logs are infrequently accessed but, when they are needed, they need to be there, they absolutely have to be readable, and they must have been stored securely.


But when cold objects are called for, they don’t need to be there instantly. The cold storage tier customer requirement for latency ranges from minutes, to hours, and in some cases even days. Customers are willing to give up access speed to get very low cost.  Potentially rapidly required database backups don’t get pushed down to cold storage until they are unlikely to get accessed. But, once pushed, it’s very inexpensive to store them indefinitely. Tape has long been the media of choice for very cold workloads and tape remains an excellent choice at scale. What’s unfortunate, is that the scale point where tape starts to win has been going up over the years. High-scale tape robots are incredibly large and expensive. The good news is that very high-scale storage customers like Large Hadron Collider (LHC) are very well served by tape. But, over the years, the volume economics of tape have been moving up scale and fewer and fewer customers are cost effectively served by tape. 


In the 80s, I had a tape storage backup system for my Usenet server and other home computers. At the time, I used tape personally and any small company could afford tape. But this scale point where tape makes economic sense has been moving up.  Small companies are really better off using disk since they don’t have the scale to hit the volume economics of tape. The same has happened at mid-sized companies. Tape usage continues to grow but more and more of the market ends up on disk.


What’s wrong with the bulk of the market using disk for cold storage? The problem with disk storage systems is they are optimized for performance and they are expensive to purchase, to administer, and even to power. Disk storage systems don’t currently target cold storage workload with that necessary fanatical focus on cost per capacity. What’s broken is that customers end up not keeping data they need to keep or paying too much to keep it because the conventional solution to cold storage isn’t available at small and even medium scales.


Cold storage is a natural cloud solution in that the cloud can provide the volume economics and allow even small-scale users to have access to low-cost, off-site, multi-datacenter, cold storage at a cost previously only possible at very high scale.  Implementing cold storage centrally in the cloud makes excellent economic sense in that all customers can gain from the volume economics of the aggregate usage. Amazon Glacier now offers Cloud storage where each object is stored redundantly in multiple, independent data centers at $0.01/GB/Month. I love the direction and velocity that our industry continues to move.


More on Glacier:


·         Detail Page:

·         Frequently Asked Questions:

·         Console access:

·         Developers Guide:

·         Getting Started Video:


By the way, if Glacier has caught your interest and you are an engineer or engineering leader with an interest in massive scale distributed storage systems, we have big plans for Glacier and are hiring. Send your resume to




James Hamilton 
b: /

Tuesday, August 21, 2012 4:46:33 AM (Pacific Standard Time, UTC-08:00)  #    Comments [6] - Trackback

Disclaimer: The opinions expressed here are my own and do not necessarily represent those of current or past employers.

<January 2015>

This Blog
Member Login
All Content © 2015, James Hamilton
Theme created by Christoph De Baene / Modified 2007.10.28 by James Hamilton