March 14, 2006 was the beginning of a new era in computing. That was the day that Amazon Web Services released the Simple Storage Service (S3). Technically, Simple Queuing Services was released earlier but it was the release of S3 that really lit the fire under cloud computing. I remember that day well. At the time I was General Manager of Frontbridge Technologies, a wholly owned subsidiary of Microsoft that provided cloud hosted email antispam, antimalware, and archiving services. From this experience, I felt like I understood the customer value of cloud hosted services. I knew customers loved the speed of provisioning and low cost so, in many ways, I was already a convert. I already felt pretty sure that cloud hosting was the future.
But still the Amazon Simple Storage Service announcement was an eye opener for me. The technology industry has 100s of announcements each day and I don’t look at many. For the most part, they are uninteresting. But the S3 announcement was game changing. Most startling was the cost of the service. It was nearly 2 orders of magnitude less expensive than we were currently paying for multi-data center redundant storage. But what was even more disruptive was a credit card was all that was needed to provision storage. There was no required proposal for financial approval, there was no RFP, no vendor selection process, no vendor negotiation, and no data center space need be found. I could just sign up and start working.
What was at least as notable as the low cost and ease of provisioning was that the announcement came from Amazon rather than a traditional enterprise IT player. Rather than a company that is dedicated to high margins, difficult negotiations, and sometimes even license usage audits, this service came from Amazon. What looks to be “large” margins at Amazon would have shareholders at most enterprise IT companies calling for immediate management change. This really was different. A different supplier, a different model, a low friction provisioning path, and a fundamentally different price that starts low and falls rather than escalates over time.
The S3 announcement generated industry-wide interest and wonderment on how it would be possible, for even a very high volume supplier, to not lose money on every single byte sold. I was completely captivated by the offering and ended up writing several thousand lines of code using S3 as the underlying storage system. At times it was a bit clunky, there was the odd sharp edge, but writing the app really established in my mind that this was the beginning of something big.
From deciding to write the app to it being up and running was measured in days and, after debugging and testing extensively, the end of the month rolled around and I got my Visa bill. Of course, I knew abstractly that S3 was disruptively priced but when I saw that my bill for the entire development and test of this application was $3.08, it just seemed wrong. Once development was complete I was still storing all the test data in S3 so the following month I got a bill for $0.07.
This was so game changing that wrote it up, blogged it internally where I worked, and demoed it to company leadership including the CTO and CEO. My presentation included a picture of Al Vermeulen, one of the early developers on S3, showed some of how S3 worked and, to underline my point that this really was different, I included my two AWS bills. My key point was this wasn’t just a stunt or a fun little experiment by Amazon but was really a fundamental new way of delivering infrastructure services. Storage was first but compute was to follow shortly.
I got increasingly interested in AWS and by 2007 was attending user group meetings, doing presentations at Amazon, and eventually I just gave up and joined the team in late 2008.
As a member of the AWS engineering team, my first impressions are probably best summarized as fast. Decisions are made quickly. New ideas end up in code and available to customers at a speed that just makes the pace of enterprise IT look like continental drift. In a previous role, I remember (half) jokingly saying “we ship twice a decade whether customers need it or not.” Now new features are going out so frequently they are often hard to track.
Another interesting aspect of AWS is how product or engineering debates are handled. These arguments come up frequently and are as actively debated at AWS as at any company. These decisions might even be argued with more fervor and conviction at AWS but its data that closes the debates and decisions are made remarkably quickly. At AWS instead of having a “strategy” and convincing customers that is what they really need, we deliver features we find useful ourselves and we invest quickly in services that customers adopt broadly. Good services become great services fast.
In some of my past roles, I’ve seen these healthy debates become bigger than life and end up dragging on unproductively for years. At AWS they are resolved in days with customer usage data and the focus swings quickly to execution. It’s really refreshing to have the normal debate to delivery equation turned upside down. Most of the effort at AWS ends up in customer hands whereas, at many jobs I’ve held, much of the effort is in bringing these competing internal efforts to resolution. The AWS speed of delivery is great for customers and I find it an exciting environment for engineers.
The best proof of innovation is customer commitment and, without a doubt, the highest form of customer commitment is to decide to run the entire company on cloud infrastructure. Netflix was the first to publically make the decision to go 100% cloud. Some of my favorite examples of cloud “all in” corporate commitments:
- 2010 Netflix
- 2013 Kempinski Hotels
- 2013 Suncorp Group
- 2014 Infor
- 2014 Nippon Express
- 2014 Notre Dame University
- 2014 National Democratic Institute
- 2015 The Guardian Media Group
From my perspective, the companies that have decided to go with cloud only infrastructure are the most interesting and the most compelling examples of the massive change sweeping the industry. But, as an example of the AWS speed of innovation, check out some of the highlights over the last decade:
2006
- 2006 Amazon S3
- 2006 Amazon SQS
- 2006 Amazon EC2
2007
- 2007 AWS introduces commerce platform for AWS, rapidly accelerating adoption, Amazon FPS
- 2007 Amazon S3 in EMEA
- 2007 Amazon Simple DB
2008
- 2008 pricing plan for Amazon SQS and new WSDL (1st price drop)
- 2008 Elastic IP Addresses
- 2008 Availability Zones for EC2
- 2008 AWS adds Premium (enterprise) support
- 2008 AWS Lowers Data Transfer Costs
- 2008 Amazon Elastic Block Storage
- 2008 EC2 SLA and GA, EC2 for Windows, EC2 for SQL Server
- 2008 New Tiered Pricing for Amazon S3
- 2008 Amazon CloudFront
- 2008 EC2 in Europe (Ireland)
2009
- 2009 AWS Management Console
- 2009 New lower pricing tiers for Amazon CloudFront
- 2009 Amazon FPS
- 2009 Elastic MapReduce
- May 2009 Elastic Load Balancing,
- May 2009 Amazon Autoscaling
- May 2009 Amazon CloudWatch
- 2009 AWS Virtual Private Clou
- 2009 New lower prices for Amazon EC2 Reserved Instances
- 2009 New lower prices for Windows instances with authentication services
- 2009 AWS enters the database market with Amazon Relational Database Service
- 2009 Lower Amazon EC2 on demand pricing
- 2009 EC2 Spot Instances
- Dec 2009 AWS S3 Pricing reductions, Free Inbound Data Transfer until 6/30/2010
- 2009 Amazon CloudFront Streaming
- 2009 AWS US West Region
2010
- 2010 Lower Pricing for Outbound Data Transfer
- 2010 AWS launches first region in Asia Pacific (Singapore)
- 2010 Amazon SNS
- May 2010 Multi Availability Zones for RDS
- May 2010 Amazon S3 Reduced Redundancy Storage
- 2010 AWS Import/Export
- 2010 Amazon CloudFront adds HTTPS Support, lowers prices, Opens NYC edge location
- 2010 Cluster Compute Instances for EC2
- 2010 AWS Identity and Access Management
- 2010 Oracle certifies enterprise software on AWS
- 2010 New lower prices for High Memory Double and Quadruple XL instances\
- 2010 Read Replicas, Lower High Memory DB Memory Instance Prices for Amazon RDS
- 2010 Amazon S3 lowers storage prices
- 2010 Amazon Route 53
- 2010 Mobile SDKs for AWS
2011
- 2011 Amazon SES
- 2011 AWS New Premium Support Plans, Lowers usage prices by 50% on existing plans
- 2011 AWS Elastic Beanstalk
- 2011 AWS CloudFormation
- 2011 AWS Tokyo Region
- May 2011 SAP certifies enterprise software on AWS
- May 2011 Amazon CloudWatch custom metrics, lower prices for Amazon EC2 monitoring
- 2011 AWS new Data Transfer Pricing
- Aug 2011 Amazon ElastiCache
- Aug 2011 AWS enables enterprises to connect their data centers directly to AWS via AWS Direct Connect
- Aug 2011 AWS launches Region dedicated to US Government Agencies and Contractors (AWS GovCloud)
- 2011 AWS Route53 lowers pricing for hosted zones
- 2011 AWS US West Region (Oregon) 100% Carbon Free Power
- 2011AWS enters South America with region in Sao Paulo, Brazil
- 2011 Amazon Elastic MapReduce support for cc2.8xlarge and reduced pricing for cc1.4xlarge
2012
- 2012 AWS Storage Gateway
- 2012 Dynamo DB
- 2012 Amazon SWF
- 2012 Amazon S3 lowers prices for standard storage
- March 2012 New lower pricing for Amazon EC2, RDS and ElastiCache
- April 2012 Amazon CloudSearch
- 2012AWS Marketplace for third-party selling of applications to businesses
- 2012 AWS Support Expands Free Tier, Adds Features, Lowers Prices
- 2012 AWS Sydney Region
- July 2012 EC2 High I/O Instances
- 2012 Amazon Glacier
- 2012 EBS Provisioned IOPs Announced
- 2012 Second Gen Standard Instances for Amazon EC2 and a price reduction for M1 Instances
- 2012 More than 6K attend first AWS user conference, re:Invent
- 2012 AWS announces Amazon RedShift
- 2012 Amazon CloudSearch free trial program and price reduction
- 2012 Amazon RDS and Amazon ElastiCache Lower Prices
- 2012 Amazon S3 Lower prices for Standard Storage and RRS
- 2012 AWS Data Pipeline
2013
- 2013 Amazon Elastic Transcoder
- 2013 High Memory Cluster Instances
- 2013 EC2 Price Reduction, global expansion of M3 Standard Instances, reduced data transfer pricing
- 2013 AWS OpsWorks
- 2013 IBM protest reveals that AWS won $600M cloud contract with CIA
- 2013 Amazon RDS reduces price of Multi-AZ Deployments
- 2013 Amazon SQS and SNS lower prices and expand free tiers – 50% price drop for SQS
- 2013 AWS CloudHSM
- 2013 AWS introduces global developer training and certification program
- 2013 Lower prices on Amazon EC2 Reserved Instances, Amazon DB
- 2013 AWS lowers prices for Amazon S3 request pricing, Windows on-demand EC2 instances by up to 26%
- May 2013 AWS first major cloud provider to gain FedRAMP certification
- 2013 AWS lowers prices of on-demand and reserved RDS instances by up to 28%
- July 2013 AWS price reductions on EC2 dedicated instances
- 2013 AWS Gartner estimates that AWS customers are deploying 5x more infrastructure on AWS than the combined adoption of the next 14 providers
- 2013 Amazon AppStream launched
- 2013 Amazon WorkSpaces launched
- 2013 EC2 GPU Instances
- 2013 price reduction for M3 Instances
- 2013 – Amazon introduces Amazon Kinesis
- 2013 Amazon EC2 HI1 Instance price reduction and Spot availability
- 2013 Amazon China Region
2014
- 2014 New Amazon EC2 M3 Instance Sizes and Lower Prices for Amazon S3 and Amazon EBS
- 2014 AWS Storage Gateway price reduction
- 2014 Amazon Redshift new SSD-based node type
- 2014 General availability for Amazon AppStream and Amazon WorkSpaces
- 2014 AWS price reduction EC2, RDS, S3, ElastiCache, and Elastic MapReduce (price reduction #42)
- 2014 AWS receives Department of Defense-Wide provisional authorization for all U.S. Regions
- 2014 Availability of R3 instances
- May 2014 Launch of AWS Management Portal for vCenter
- May 2014 Introducing Amazon EBS encryption
- 2014 AWS availability of a new SSD-backed volume type for Amazon EBS (price drop #43)
- 2014 AWS Opened Pop-up Loft in San Francisco on temporary basis
- 2014 Amazon Redshift free trial and price reductions in Asia Pacific (price drop #44)
- 2014 new low-cost general purpose instance type for Amazon EC2
- 2014 Introduction of Amazon Zocalo (now known as Amazon WorkDocs)
- 2014 Route 53 price reduction (price drop #45)
- 2014 Introduction of services for mobile developers: Amazon Cognito, Amazon Mobile Analytics, AWS Mobile SDK, and Amazon SNS Mobile Push
- 2014 Launched Amazon CloudWatch logs
- 2014 AWS GovCloud achieves Department of Defense CSM Level 3-5 Provisional Authorization
- 2014 General availability of Zocalo (now known as Amazon WorkDocs)
- 2014 AWS EU (Frankfurt) region
- 2014 AWS Reopened Pop-up Loft in San Francisco on permanent basis
- 2014 Introduction of native support for document models like JSON into DynamoDB
- 2014 Introduction of AWS Directory Service
- 2014 AWS achieves ISO- 9001 certification
- 2014 Expansion of APN Partner benefits, introduction of new Managed Service and SaaS Partner Programs, and expansion of APN Partner training
- 2014 CloudSearch price drop (price drop #46)
- 2014 Introduction of Amazon Aurora, a MySQL-compatible database
- 2014 Enterprise security & governance services: Key Management, AWS Config, & AWS Service Catalog
- 2014 New application lifecycle management services introduced: AWS CodeDeploy & AWS CodePipeline
- 2014 Amazon EC2 Container Service introduced
- 2014 AWS Lambda announced
- 2014 Amazon EC2 C4 and EBS pre-announced
- 2014 AWS pledges long-term commitment to achieve 100 percent renewable-energy usage
- 2014 Data transfer and CloudFront price drop (price drop #47)
2015
- 2015 Amazon Wind Farm Fowler Ridge Announced
- 2015 Amazon WorkMail launches in preview
- 2015 Amazon Machine Learning, fully managed service announced
- 2015 AWS Marketplace for Desktop Apps and Amazon WorkSpaces Application Manager
- 2015 ISVs Go “All-In” with AWS announcement: MicroStrategy, Software AG, TIBCO, and Onshape
- May 2015 AWS Educate to Accelerate Cloud Learning in the Classroom
- 2015 Amazon Solar Farm US East Announced
- 2015 M4 Instances for Amazon EC2
- 2015 AWS Opens Second Global “City on a Cloud Innovation Challenge”
- 2015 AWS 2016 India region pre-announced
- 2015 AWS Opened Pop-up Loft in New York City
- 2015 ISVs Go “All-In” with AWS announcement: Looker, Qlik, Sumo Logic, and Works Applications.
- 2015 AWS Announces Amazon API Gateway
- 2015 AWS Announces AWS Device Farm
- 2015 Amazon Wind Farm US East Announced
- 2015 Amazon Aurora General Availability
- 2015 AWS Announced Pop-up Lofts opening in London and Berlin
- 2015 AWS Announced Amazon QuickSight at re:Invent
- 2015 AWS Announced AWS Snowball and Amazon Kinesis Firehose at re:Invent
- 2015 AWS Announced AWS Database Migration Service and Amazon RDS for MariaDB at re:Invent
- 2015 AWS and Accenture Announce the Accenture AWS Business Group
- 2015 AWS Announced AWS IoT preview at re:Invent
- 2015 AWS Pre-Announced the AWS UK region to be third in the European Union
- 2015 Amazon Wind Farm US Central Announced
- 2015 AWS Announced IoT General Availability
- 2015 AWS Introduced t2.nano, the smallest and lowest cost Amazon Ec2 instance
2016
- 2016 AWS launched Korea region as fifth in Asia Pacific (Seoul)
- 2016 Amazon WorkMail General Availability
- 2015 AWS pre-announced Canada-Montreal region
- 2016 AWS Announced Amazon Lumberyard and Amazon GameLift Availability for Game Developers
- 2016 AWS Announced the AWS Pop-up Loft in Tel Aviv
9-22-2017
Hi!
In your opinion James what most likely would handicap cloud services world wide impacting participants vital storage??
Greg, you’ve asked several times about cloud computing security and cloud computing outages. I sense I’m not answering your question fully. Here you are asking about really is “what most likely would handicap cloud services worldwide” — this means that data centers spread all over the world with many facilities on all continents, with independent power, redundant backup power, redundant network links, all had some global issue at the same time. I really struggle to come up with anything that could cause such an event. It’s possible that a poorly designed network could encounter short term interruptions but these should be incredibly unlikely in reasonable designs. If you are really asking about data loss when you say “participants vital storage”, I suspect that is less likely than multiple alien races all arriving on earth to meet us on the same day. Possible, but very close to the least likely mishap that could impact an individual.
Hi James, I remember you mentioned that the public cloud providers (AWS/Azure/GAP etc) also have the top security talents & best capabilities in industry. Does it mean that most traditional security companies will fade away or lose significant Revenue in the cloud world? 2. Do you think next-generation security start-ups such as SecurityScorecard still have a chance to compete with these cloud giants? If yes, in which area? thanks
The major cloud providers are deeply invested in security and are doing good work. But no matter how much is invested, there are always opportunities for innovative startups. Intuitively, you would think that once a technology area becomes important and gets deep investment, it would no longer be possible to contribute. But, the opposite happens. When a technology area becomes important, even small gains are super valuable and the opportunities for startups are still there and sometimes even better.
The more important a given area becomes, the more value even small improvements will be. Security startups have massive opportunity and I expect to continue to see big exits even though it’s hard to predict in what exact security area these innovations will focus.
I am also a engineer for public cloud service. AWS, as the leader of industry, does a great job. but I wonder that why we do this a lot of feature after reading this post. So, I read all your paper in http://mvdirona.com/jrh/work/, and I have a summery in https://docs.google.com/document/d/1-AqnUucqv_skPfSLYM-BYvmg8VLKuJqBYMjohRkG3zE/edit . It is a great job, and a brave one.
It is definitely a fun time to be working in cloud computing. We’re both lucky. Thanks for all the hard work to bring that summary together.
Hello James, I loved the slide in your 2014 Re:Invent Innovation at Scale presentation where you talked about the server capacity AWS added each day in 2014. Do you have an updated statistic for 2016?
Thanks
That’s a great suggestion Todd. It’s got to be even bigger than it was in 2014 given the current growth rate. I’ll plan to get current scaling data computed and it’ll be part of my talk Tuesday night at 8pm during re:Invent. I’m hoping to show a quick look inside one of our facilities as well. Thanks for the suggestion.
Yay, time to play spot the difference again. The pie chart has not changed in a while.
Getting some peeks into AWS would be nice. Will we recognize some of the things that were very small pictures in your older talks? Fun times for outside observers. However that turns out, I’m certain that it won’t be a repeat of the 2015 generic stock photo talk someone else did.
What will it be, radically different or radically boring? I’m thinking about the old talk about hardware & temperatures and what FB and G showed about some of their DCs. Then back to that “industry wide data” pie chart, and what if you were able to simply remove a category…
Apropos spotting the difference. In 2014 the fiber count in the example region was 82864, unchanged in the 2015 talk. Still the same count in 2016? If so maybe spend a moment to explain that. =)
Your standards are high but I’ll certainly try to make sure it has lots of new data. You specifically asked about the fiber count. It has exploded over the last two years since that number was last produced. Pretty much every thing has — rough numbers, we’re 2x the size from the last time I did a talk at re:Invent. Not sure what data I’ll get out there this year but I promise it’ll all be up to date and I suspect you’ll find some of it interesting. See you Tuesday night at 8PM during re:Invent.
Wonderful summary, indeed so many features have been delivered. we’re very happy AWS users and our startup wouldn’t exist without AWS. but the competition is using the trail-blazed path to catch up and there are still more innovations to deliver. It’s interesting to see how some of the most powerful features that leave the competition behind, have been delivered as early as 2009…
Another thing that AWS would possibly need to play catch up to are higher-level services like ‘Marketing Clouds’ or Google’s vision-related APIs. I think AWIS (Amazon Alexa) is one such untapped resource and I think AWS should deepen its GPU leadership by extending the offerings.
I agree with you Gill that it’s still very early days in the cloud industry and there will be lots of change comming. I also agree that AWS competitors do, in some cases, get to fast follow and, in effect, get to cover some of the ground covered by AWS faster. Knowing that a managed relational database service is needed and roughly what a successful one might look like is certainly a time advantage. But, for many characteristics of a cloud service, stealing a quote from AWS leadership, there really is no compression algorithm for experience.
One of the big players just went through 20 min outage that impacted multiple regions. That means if you were using that service and had been smart and made your application, multi-data center and even multiple region redundant, you had a good chance of being down for 20 min. Learning to run well and reliably takes time. As proof of that, there were some early outages at AWS and some were big — fortunately, we never went down across multiple multiple data centers in multiple regions.
It takes time to learn to do offer an enterprise quality service while at the same time constantly adding new services and improving the underlying designs. It can be done and there certainly will be several competitors that do it but it won’t “just happen” because AWS has already done it. All leaders in the cloud will have had to learn some fo these lessons and some will perhaps need to learn some them multiple times. Delivering an operationally rock solid service is challenging.
However, I agree with your core point: it’s very early in cloud computing, fast following competitors are moving quickly, and the the next few years are going to be super interesting, some very good work will be done, and both the services offered and how well they are offered are going to be improving fast at all cloud competitors including AWS. It’s going to be an exciting time.
James, love this post. I know you were writing about success of AWS but the two paragraphs on “debates” were priceless. We have one of those going on right now. So, I just quoted you and pasted them right in to the debate thread.
And for that I owe you a beer next time I am in Seatle or where ever you are floating these days!
Hey Leon, good hearing from you. Debate is like cooking spices. A bit is good but endless amounts don’t work at all. Debate that gets different ideas on the table is good. Debate that reveals new facts is again good. But, fairly quickly, all the value from debate is had and it’s just slowing you down.
In most industries, and especially in tech, there are few assets more valuable than engineering speed.
Hey James, its been a while! Wonderful post summarizing the accomplishments. I can certainly appreciate the “speed of moving from debate to execution”, and the “speed of delivery” aspects, having been in similar situations, and agree that these are key to be successful in this space.
All the best. Let’s catch up some time.
It has been a very long time indeed. Next time I’m through the Valley.
James,
This is a fantastic post summarizing the drivers underlying successful infrastructure services innovations, with AWS as a case in point. “Good services” are the ones where customers intuitively discern their value (like you did with your credit card bills for S3!), and not surprisingly, become “great services”. If there is a takeaway from this post, it is that successful innovation is an equation comprising (1) value to customers, (2) continuously develop and deploy new technology and (3) a culture reinforcing the above two.
This is a great complement to Werner’s “10 lessons from 10 years” post. Thanks for sharing your thoughts!
Thanks Nachiketh. It’s been a fun 10 years and we’re just getting started.
I think you meant to say “It’s still Day 1” :)
Also, seems like mTurk always gets the short end of the stick on these posts. I remember Jeff Barr sharing it w/ folks at the first Seattle Mindcamp back in winter 2005.
Mechanical Turk a cool service that continues to grow but we don’t really think of it as a part of AWS and don’t operate it that way.