I forget what brought it up but sometime back Sriram Krishnan forwarded me this article on Mike Burrows and his work through Dec, Microsoft, and Google (The Genius: Mike Burrows' self-effacing journey through Silicon Valley).  I enjoyed the read.  Mike has done a lot over the years but perhaps his best known works of recent years are Alta Vista at DEC and Chubby at Google.

I first met Mike when he was at Microsoft Research.  He and Ted Wobber (also from Digital) came up to Redmond to visit.  Back then I led the SQL Server relational engine development team which included the full text search index support.   I was convinced then, and still am today, that relational database engines do a good job of managing structured data but a poor job of the other 90 to 95% of the data in the world that is less structured.  It just seems nuts to me that customers industry-wide are spending well over $10B a year on relational database management systems and yet only being able to effectively use these systems to manage a tiny fraction of their data.  As an increasing fraction of the structured data in the world is already stored in relational database managements systems, industry growth will come from helping customers manage their less structured data. 

To be fair, most RDMBS (including SQL Server) do support full text indexing but what I’m after is deep support for full text where the index is a standard access method rather than a separate indexing engine on the side and, more importantly, full statistics are tracked on the full text corpus allowing the optimizer to make high quality decisions on join orders and techniques that include full text indices.

If you haven’t read Mike’s original Chubby paper, do that: http://labs.google.com/papers/chubby.html.  Another paper is at: http://labs.google.com/papers/paxos_made_live.html. Chubby is an interesting combination of name server, lease manager, and mini-distributed file system.  It’s not the combination of functionality that I would have thought to bring together in a single system but it’s heavily used and well regarded at Google.  Unquestionably a success.

                                --jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

The years of Moore’s law growth without regard to power consumption are now over. On the data center side, power isn’t close to the largest cost of running a large service but it is one of the largest controllable costs and it has been in the press frequently of late.  On the client side, battery power is the limiting factor. 

It is worth understanding what devices consume the most power since most laptops provide some form of user control.   Most systems allow LCD backlight dimming, the CPU power consumption can be lowered (a combination of factors including reducing clock speed and voltage), wireless radios can be switched off, and disks activity can be curtailed or eliminated.  Where does the power go? 

The data below was measured by Mahesri and Vardhan with an Thinkpad R40 as the system under test:

 Data from: http://www.crhc.uiuc.edu/~mahesri/classes/project_report_cs497yyz.pdf.

The dominant consumer by a significant factor is the CPU.   This power consumption is, of course, very load dependent particularly in multi-core systems where the spread between minimum and maximum power dissipation is even higher. The second largest consumer is the LCD backlight, which isn’t surprising.  Two LCD-related findings that I did find surprising: 1) the LCD without backlight is a very light consumer of power, and 2) there is a perceptible difference in power consumption between mostly black and mostly white backgrounds.   The hard disk drive power consumption was notably less than I expected with only 2.8W dissipated during active reading.

I wrote up more detail in: ClientSidePower6_External.doc (130 KB).

                                                 --jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

My rough notes from the Web 2.0 Keynote by Yahoo! CTO Ari Balogh:

·         Yahoo! is making three big bets:

1.       Be the starting point for all consumers

2.       Be the must buy for advertisers

3.       Provide an Open Platform

·         Focus of today’s talk is on the later, open platform.

·         Yahoo! broad set of assets are well known

·         We lead in 7 areas including: Mail, My Front Page and Messenger (the full list was provided nor how Yahoo! was computed to “lead” in these area)

·         350M unique users/month and 500M users overall

·         20B page views/month

·         250M users minutes per month

·         10B user relationships across properties and this is the real asset

·         Yahoo! has been open since 2003

·         25+ APIs

·         200K App IDs (hints at the large number of developers)

·         #2 API in the world with Flikr

·         1B UI files/served/week

·         Y!OS: (Yahoo! Open Strategy)

·         Announcing today they are open all assets at Yahoo! to developers

·         Planning to make all experiences at Yahoo “social”

·         Provide an open developer platform with hooks for third parties to make experiences more social

·         Built into application platform:

·         Security: give users control of their data.  Where they want to share what with who.

·         Application gallery. A common way to <JRH>

·         Unify profiles across all of Yahoo (this will take a while) and provide access to developers the social graph and the notification engine. Open up developer access to produce events and the platform includes the ranking engine to show users the most relevant events based upon their context (including social graph).

·         Making Yahoo! more social:

·         Not creating another social network

·         Making all of yahoo “social”

·         “social” isn’t a destination but rather a dimension of a user experience

·         “social” drives relevance, community, and virality

·         Showed some examples:

·         Email client showing messages most relevant on the basis of social network

·         Same basic idea for a “My Yahoo!” page

·         When?

·         Search Monkey is the first step

·         Later this year they will deliver Y!OS and provide more uniform and consistent developer access

·         Making Yahoo! more social will take longer with property by property steps being taken over next few years

·         Summary:

1.       Rewiring Yahoo! from the ground up

2.       Open Yahoo! to developers like never before

3.       Making Yahoo! more social across Yahoo! properties and to third party developers

The 12 min presentation is at: Ari Balogh Web 2.0 Expo Keynote.

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

Flash SSDs in laptops have generated considerable excitement over the last year and are in use at both extremes of the  laptop market.  At the very low end, where only very small storage amounts can be funded, NAND Flash is below the below the disk price floor.  Mechanical disks with all their complexity are very difficult to manufacture for less than $30 each.  What this means is that for very small storage quantities, NAND Flash storage can actually be cheaper than mechanical disk drives even though the price per GB for Flash is larger. That’s why the One Laptop Per Child project uses NAND flash for persistent storage.  At the high end of the market, NAND flash considerably more expensive than disk but, for the premium price, offers much higher performance, more resilience to shock and high G handling, and longer battery life.

Recently there have been many reports of high-end SSD laptop performance problems.  Digging deeper, this is driven by two factors: 1) gen 1 SSDS produce very good read performance but aren’t particularly good on random write workloads, and 2) performance degradation over time.  The first factor can be seen clearly in this performance study using SQLIO: http://blogs.mssqltips.com/blogs/chadboyd/archive/2008/03/16/ssd-and-sql-sqlio-performance.aspx.  The poor random write performance issue is very solvable using better Flash wear leveling algorithms, reserving more space (more on this later), and capacitor backed DRAM staging areas. In fact STEC ZeusIOPS is producing great performance numbers today, Fusion IO is reporting great numbers, and many others are coming.  The first problem, that of poor random write performance, can be solved and these solutions will migrate down to the commodity drives. 

The second problem, the performance degradation issue, is more interesting.  There have been many reports of laptop dissatisfaction and very high return rates: Returns, technical problems high with flash-based notebooks. Dell has refuted these claims Dell: Flash notebooks are working fine but there are lingering anecdotal complaints of degrading performance. I’ve heard it enough myself that I decided to dig deeper.  I chatted off the record with an industry insider on why SSDs appear to degrade over time.  Here’s what I learned (released with their permission):

On a pristine NAND SSD made of quality silicon to ensure write amplification remaining at 1 [jrh: write amplification refers to the additional writes that are caused by a single write due to wear leveling and the Flash erase block sizes being considerably larger than the write page size – the goal is to get this as close to 1 as possible where 1 is no write amplification], given a not-so-primitive controller and reasonable over-provisioning (greater than 25%), a sparsely used volume (less than half full at any time) will not start showing perceptible degraded performance for a long time (perhaps as long as 5 years, the projected warranty period to be given to these SSD products).

If any of the above conditions is changed, the write amplification will quickly degrade ranging from 2 to 5, or even higher.  That contributes to the early start of perceptible degraded write performance.  That is, on a fairly full SSD you’d start having perceptible write performance problems more quickly, and so on.

Inexpensive (cheap?) SSD made of low-quality silicon will likely to have more read errors.  Error correction techniques will still guarantee correct information being returned on reads.  However, each time a read error is detected, the whole “block” of data will have to be relocated elsewhere on the device.  A not-so-well designed controller firmware will worsen the read delay, due to poorly implemented algorithms and ill-conceived space layout that take longer to search for available space for the relocated data, away from the read error area.

If the read-error-data-relocation happens to collide with the negative conditions that plague the write performance above, you’d start seeing overall degraded performance very quickly.

Chkdsk may have contributed to the forced relocation of the data away from where read errors occurred, hence improving the SSD performance (for a while) until the above collisions happen.  Perhaps the same when Defrag is used.

In short, performance degradation over time is unavoidable with SSD devices.  It’s a matter of how soon it kicks in and how bad it gets; and it varies across designs.

We expect the enterprise class SSD devices to be as much as 100% over-provisioned (e.g., a 64GB SSD actually holds 128GB of flash silicon). 

Summary: there are two factors in play. The first is that SSD write random performance is not great on low end parts so ensure you understand the random write I/O specification before spending on an SSD. The second one is more insidious in that, in this failure mode, the performance just degrade slowly over time.  The best way to avoid this phenomena is to 2x over-provision.  If you buy N bytes of SSD, don’t use more than ½N and consider either chkdsk or copying the data off, bulk erasing, and sequentially copying back on . We know over-provisioning is effective. The later techniques are unproven but seem likely to work. I’ll report supporting performance studies or vendor reports when either surface.

                                                --jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

It’s not often I come across three interesting notes in the same day but here’s another. Earlier today the Jim Gray Systems Lab was announced and it will be lead by long time database pioneer David DeWitt.  This is great to see for a large variety of reasons. First of all it’s wonderful to see the contribution of Jim Gray to the entire industry recognized in the naming of this new lab.  Very appropriate.  Second I’m really looking forward to working more closely with DeWitt.  This is going to be fun.

This is “earned” in that Madison has been contributing great database developers to the industry for what seems like forever – I’ve probably worked with more Madison graduates over the years than any other single school. It’s good to see a systems focused research lab opened up there. 

It’s also good to see this project come together. I was involved in earlier discussions on this project some years back and, although we didn’t find a way to make it happen then, I really liked the idea.  I’m glad others were successful in doing the hard work to get this project to reality.

·         University of Wisconsin at Madison News: http://www.news.wisc.edu/15097  

·         DeWitt Interview (from above): http://www.microsoft.com/presspass/features/2008/apr08/04-23DeWitt.mspx

·         Server and Tools Business News Blog: http://blogs.technet.com/stbnewsbytes/archive/2008/04/23/an-addition-to-the-sql-server-team.aspx

·         Information Week: http://www.informationweek.com/news/software/database/showArticle.jhtml;jsessionid=2PMY2VDAXNZHSQSNDLOSKHSCJUNN2JVN?articleID=207401497

                                                --jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

Earlier today, Amazon AWS announced a reduction in egress charges.  The new charges:

·         $0.100 per GB - data transfer in

·         $0.170 per GB - first 10 TB / month data transfer out

·         $0.130 per GB - next 40 TB / month data transfer out

·         $0.110 per GB - next 100 TB / month data transfer out

·         $0.100 per GB - data transfer out / month over 150 TB

Compared with the old:

·         $0.100 per GB - data transfer in

·         $0.180 per GB - first 10 TB / month data transfer out

·         $0.160 per GB - next 40 TB / month data transfer out

·         $0.130 per GB - data transfer out / month over 50 TB

Most networking contracts charge symmetrically for ingress and egress – you pay the max of the two -- so the ingress cost to Amazon is effectively zero.

Note that it’s a non-linear reduction favoring higher volume users.  TechCrunch reported a couple of days back that the Amazon AWS customer base has rapidly swung from a nearly pure start-up community to more of a mix of startups and very large enterprises with the enterprise customers now bringing the largest workloads (http://www.techcrunch.com/2008/04/21/who-are-the-biggest-users-of-amazon-web-services-its-not-startups/).  Not really all that surprising – I expected this to happen and talked about it in the Next Big Thing. What is surprising to me is the speed with which the transformation is taking place. I was predicting workload mix shift to happen at AWS 3 to 5 years from now. Things are moving quickly in the services world.

                                                --jrh

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

Live Mesh has been under development for a couple of years now.  Now it’s hear in “technology preview” form. I think the first public mention was probably back in March of last year in a blog entry by Mary Jo Foley that mentioned Windows Live Core (http://blogs.zdnet.com/microsoft/?p=349). Last night Amit Mital, General Manager of Windows Live Core, did a blog entry that coves Live Mesh in more detail that previously seen: http://dev.live.com/blogs/devlive/archive/2008/04/22/279.aspx.

UPDATE: The report above attributing first mention of Windows Live Core to Mary Jo Foley was incorrect.  The sleuths at LiveSide appear to have reported this one first: http://www.liveside.net/blogs/main/archive/2007/03/25/windows-live-core.aspx.

Live Mesh is a platform that supports synchronizing data across devices, a platform for deploying  and managing apps that run on multiple devices, supports screen remoting making all your devices and applications available from anywhere, and it strikes an interesting balance exploiting both cloud services supported features and unique device capabilities. The initial device support is Windows only but Mac and other device clients are coming as well.

Screen shots are up on CrunchBase: http://www.crunchbase.com/product/windows-live-mesh.

Ray Ozzie did a 36 min Channel 9 interview with Jon Udell: http://channel9.msdn.com/showpost.aspx?postid=399578.

Abolade Gbadegesin, Live Mesh Architect, did a video on Live Mesh Architecture that is worth checking out: http://channel9.msdn.com/Showpost.aspx?postid=399577.

Demo video: http://www.on10.net/blogs/nic/Hands-on-with-Live-Mesh/.

                                                --jrh   

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

Here’s a statistic I love, Facebook is running 1,800 MySQL Servers with only 2 DBAs. Impressive. I love seeing services show how far you can go towards admin-free operation. 2:1,800 is respectable and for database servers it downright impressive. This data from a short but interesting report at: http://www.paragon-cs.com/wordpress/?p=144.

The Facebook fleet has grown fairly dramatically of late.   For example, Facebook is the largest Memcached installation and the most recent reports I had come across have 200 Memcached servers at facebook.  At the Scaling MySQL panel, they report 805 Memcached servers.

1,800 MySQL Servers, insulated by 805 Memcached servers, and driven by 10,000 web servers. Smells like success.

                                                --jrh

Thanks to Dare Obasanjo for pointing me to this one.

James Hamilton, Windows Live Platform Services
Bldg RedW-D/2072, One Microsoft Way, Redmond, Washington, 98052
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | JamesRH@microsoft.com

H:mvdirona.com | W:research.microsoft.com/~jamesrh  | blog:http://perspectives.mvdirona.com

Back in March I speculated that Google was soon to announce a third party service platform. Well, on the evening of April 7^th, Google Application Engine was announced.  It’s been heavily covered over the last couple of weeks and I’ve been waiting to get a beta account so I can write some code against it. I’ve not yet got an account but Sriram Krishnan has been playing with it and sent me the following excellent review.

·         Guest book development video: