FlockDB

Nicholas Audo | @naudo

1
 What?

• distributed, fault tolerant “graph database”

• Twitter, so you know it’s reliable

• optimized for large adjacency lists, fast reads &

writes, & page-able set queries.

• not optimized for graph-traversal operations

• solution to a fairly specific problem

2
 What? (cont.)

• 13 billion edges

• sustain 20k/second writes & 100k reads/second

• scales horizontally

• low-latency, high throughput environments

3
 Graph what?

• nodes, edges for info

• node - object in a graph database

• edge - connect 2 data elements

• neo4j

4
 Smörgåsbord of Tech

• Java 1.6+,
• Scala
• Ruby 1.8
• MySQL 5+
• thrift 0.2
5
 Features
• Sharding

• Thrift interface

• “Transactions”

• State

• Compound queries

• Result paging

• “Migrations”

6
 Thrift Interface
• Apache incubator
• Polyglot
• C++, Java, Python, PHP, Ruby, Erlang,
Perl, Haskell, C#, Cocoa, Smalltalk, and
OCaml
• Ruby client
7
 “Transactions”

• Not real transactions

• Bundle RPC calls
• Mass actions

8
 Transactions - Code
>> flock.transaction do |t|
>> t.add(1, :follows, 20)
>> t.add(1, :follows, 30)
>> end
=> nil

Mass Actions
>> flock.remove(29, :follows, nil)

9
 State

• Nothing is deleted from MySQL

• Performance
• “removed” - similar to deleted
• “archived” - hides from normal queries

1
 State - Code

• flock.add(1, :follows, 20)

• flock.archive(1, :follows, nil)
• flock.unarchive(1, :follows, nil)
• flock.remove(1, :follows. nil)

1
 Simple Queries

>> flock.select(nil, :follows, 1).to_a

=> [22, 21, 20]

1
 Compound Queries

• Not that much more complicated

• intersect
• union
• difference

1
 CQ - Code
• Ex, Do I follow anyone that follows Justin Bieber?
>> flock.select(1, :follows, nil).intersect(nil, :follows, 20).to_a

=> []

• Union
>> flock.select(1, :follows, nil).union(nil, :follows, 1).to_a

=> [2,3,47]

• What if I just want the sexy spam bots that follow

me?
>> flock.select(nil, :follows, 1).difference(1, :follows, nil).to_a

=>[3,4,7]
1
 Result Paging

>> p = flock.select(1, :follows, nil).union(nil, :follows, 1).paginate(2)

=> #<Flock::Operation:0x10157a538 ...>

>> p.next_page

=> [30, 25]

1
 Migrations

• Data, not schema

• Run command, confrim
flocker.rb -D migrate-one 508882612 127.0.0.1

1
 Install FlockDB

• git clone http://github.com/twitter/flockdb.git

• cd flockdb; ant -Dskip.test=1

• sbt “soon”

1
 How to use
• Setup some environment variables
export DB_USERNAME=”root”
export DB_PASSWORD=”password”

• Run setup-env.sh
• kills / restarts flock
• creates dev database
• runs flocker to create shards
• Ask where a shard is stored
./src/scripts/flocker.rb -D find 1 --graph 1

1
 Under the Hood

• graphs stored as sets of edges between nodes

• identified by 64-bit integers

• 64-bit position for sorting

 More

• http://engineering.twitter.com/2010/05/introducing-flockdb.html

• http://github.com/twitter/flockdb

• http://github.com/twitter/flockdb-client/tree/

• http://en.wikipedia.org/wiki/Graph_database

• http://incubator.apache.org/thrift/

• http://github.com/twitter/gizzard