My notes from an older talk done by Ryan Barrett on the Google App Engine Data store at Google IO last year (5/28/2008). Ryan is a co-founder of the App Engine team.

·         App Engine Data Store is build on Big Table.

o   Scalable structured storage

o   Not a sharded database

o   Not an RDBMS (MySQL, Oracle, etc.)

o   Not a Distributed Hash Table (DHT)

o   It IS a sharded sorted array

·         Supported operations:

o   Read

o   Write

o   Delete

o   Single row transactions (optimistic concurrency control).

o   Scans:

1.       Prefix scan

2.       Range scan

·          Primary object: Entity

o   Stored in entity table

o   Each row has a name and the row name is fully qualified /root/parent/entity/child

o   Each entity has a parent or is a root entity and may have child entities

o   Primary key is the fully qualified name and this can’t change

o   An entity can’t be reparented (it can be deleted and created with a different parent)

·         Queries:

o   Queries can be filtered on kind and Ryan says kind “is like a table” (kind can be parent, child, grandparent, …)

o   Queries can be filtered on ancestor

o   Query language is GQL (presumably Google Query Language) which is a small subset of SQL

o   All queries must be expressible as range or prefix scans (no sort, orderby, or other unbounded size operations supported)

·         Secondary index implementation:

o   Indexes are also implemented as BigTable tables

o   Kind Index:

·         Contents: (kind, key)

o   Single property index:

·         Coentents: (kind, name, value)

·         Two copies of this index maintained: 1) ascending, and 2) descending

o   Composite indexes:

·         Contents: (kind, value, value)

·         Supports multi-property indexes

·         Built on programmer request but not on use (query returns error if required doesn’t exist)

·         Programmer can specify what composite indexes are needed in index.yaml

·         SDK creates composit index specs automatically in index.yaml as queries are run

·         Entity group

o   Supports multi-entity update

·         Defined by root entity (all entities under a root are an entity group)

·         All journaling and transactions done at root

·         Text and Blobs:

o   Not indexed. All other properties are

James Hamilton, Amazon Web Services

1200, 12^th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com  

H:mvdirona.com | W:mvdirona.com/jrh/work  | blog:http://perspectives.mvdirona.com