I participated in the Self Managing Database Systems closing panel titled Grand Challenges in Database Self-Management. Also on the panel were:

• Anastassia Ailamaki (Ecole Polytechnique Federale de Lausanne, Switzerland)
• Shivnath Babu (Duke University, USA)
• Nicolas Bruno (Microsoft Research, USA)
• Benoît Dageville (Oracle, USA)
• James Hamilton (Amazon, USA)
• Sam Lightstone (IBM Toronto Software Lab, Canada)

Ken Salem of University of Waterloo (my alma mater), organized the panel and asked each of us to: “identify one substantial open problem related to self-managing databases – something that people interested in this area should be working on. Feel free to define “database” broadly.”

The panel was organized to give us each 10 min to present our grand challenge followed by Audience Q&A. As my topic, I chose: RDBMS Loosing Workloads in the Cloud. The basic premise is that very high-scale service workloads actually often do use RDBMS but they use them as simple ISAMs and full RDBMS functionality is rarely exploited. And, many data management tasks in new domains are done by MapReduce, Memcached, or other solutions. Basically, RDBMSs are heavily used in the cloud but only a tiny percentage of their features and many new workloads aren’t using RDBMS at all.

The call to action is to focus on cost. Go where the user pain is (service optimize for cost). And, as a test, if the very first thing the largest users do is shut off auto-management, the feature isn’t yet right. We should be implementing auto-management systems that the very biggest users actually chose to use. These very large customers prioritize stability over the last few percentage of optimization. They don’t want to get called in the middle of the night when a plan changes. My recommendation is to adopt a do no harm mantra and, failing that, detect and correct harm before it has broad impact. Be able to revert back a failed optimization fast. Focus on the problems where human optimization is not possible. For example resource allocation is extremely dynamic. The correct amount of buffer pool, sort heap, and hash join space varies with the workload and can’t be effectively human set. This type of problem is perfect for auto-management.

Focus on optimizations that are 1) stable (do no harm) or 2) dynamic where you can do better than a static, human chosen setting.

I would also like to the see the community define “database” to be all persistent data management rather on applications written to relational interfaces. The problem is far larger.

My slides are at: http://mvdirona.com/jrh/talksAndPapers/JamesHamilton_SMDB_Panel.pdf.

–jrh

James Hamilton, Amazon Web Services

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com

H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com