X-Tracing Hadoop: Andy Konwinski

· Berkeley student with the Berkeley RAD Lab

· Motivation: Make Hadoop map/reduce jobs easier to understand and debug

· Approach: X-trace Hadoop (500 lines of code)

· X-trace is a path based tracing framework

· Generates an event graph to capture causality of events across a network.

· Xtrace collects: Report label, trace id, report id, hostname, timestamp, etc.

· What we get from Xtrace:

o Deterministic causality and concurrency

o Control over which events get traced

o Cross-layer

o Low overhead (modest sized traces produced)

o Modest implementation complexity

· Want real, high scale production data sets. Facebook has been very helpful but Andy is after more data to show the value of the xtrace approach to Hadoop debugging. Contact andyk@cs.berkeley.edu if you want to contribute data.

ZooKeeper: Benjamin Reed (Yahoo Research)

· Distributed consensus service

· Observation:

o Distributed systems need coordination

o Programmers can’t use locks correctly

o Message based coordination can be hard to use in some applications

· Wishes:

o Simple, robust, good performance

o Tuned for read dominant workloads

o Familiar models and interface

o Wait-free

o Need to be able to wait efficiently

· Google uses Locks (Chubby) but we felt this was too complex an approach

· Design point: start with a file system API model and strip out what is not needed

· Don’t need:

o Partial reads & writes

o Rename

· What we do need:

o Ordered updates with strong persistence guarantees

o Conditional updates

o Watches for data changes

o Ephemeral nodes

o Generated file names (mktemp)

· Data model:

o Hiearchical name space

o Each znode has data and children

o Data is read and written in its entirety

· All API take a path (no file handles and no open and close)

· Quorum based updates with reads from any servers (you may get old data – if you call sync first, the next read will be current as of the point of time when the sync was run at the oldest. All updates flow through an elected leader (re-elected on failure).

· Written in Java

· Started oct/2006. Prototyped fall 2006. Initial implementation March 2007. Open sourced in Nov 2007.

· A Paxos variant (modified multi-paxos)

· Zookeeper is a software offering in Yahoo whereas Hadoop

Note: Yahoo is planning to start a monthly Hadoop user meeting.