Professional Documents
Culture Documents
Erlang in Production
Erlang in Production
Erlang in Production
in Erlang
Lessons Learned
•390 apps
•1,195 devices
•19,047 log messages
so?
•pretty big project
•very dynamic
•quickly growing in size & features
System design
Data
•graph-oriented (like Facebook)
•Riak for most data: nodes, links, streams
•etcd for consistent cases (Raft
consensus): settings, cluster structure
•in-memory ETS: cache, sync ordering
•pre-built data for reading
Graph
•nodes: id, rev, attrs, system flags
•links: from-id, to-id, type
•holds essential part of logic, i.e. session
is a link from profile to device etc
•Facebook TAO model: fetching nodes and
simplest links-walking
•implemented as independent library
K-ordering
•revision control for each entity
•to ensure all client calls are idempotent
•k-ordering for cursor-based sync (**)
•flake library (snowflake-like)
•one more, riak_id
K-ordering
** client tells server max revision
ever seen (a.k.a. cursor)
server send changed data only
(current rev > client max rev)
K-ordering
•github.com/twitter/snowflake (Scala)
•github.com/boundary/flake (Erlang) *
•github.com/seancribbs/riak_id (Erlang)
Streams
•activitystrea.ms
•Actor, Action, Object, Target
•cases: timelines, activity streams,
chats, notification center
•linked lists
•cursor-based fetch
Streams
Offline support
•cases: follow-ups, notes, cleared
messages etc
•event-sourcing (both server & clients)
•LWW for conflicted rewrites
ETS
• use to avoid state copy in gen_server
• github.com/uwiger/gproc
• github.com/wooga/etest
• github.com/wooga/etest_http
• github.com/bash/riak_kv
• github.com/basho/riak_core
• github.com/basho/riak_pipe
• github.com/basho/lager
• github.com/marccampbell/flake
• github.com/gleber/erlcloud
Questions #1
Performance
• ~20-25ms for most responses
• 100+ connections without any impact
• faster then Python & Ruby
• not as fast as Scala, Clojure and Go
• ... but do you really care?
Questions #2
Candidates
• Erlang (our choice)
• Scala (jvm)
• Clojure (jvm)
• Python (bad fit)
• Go (too large project)
• Haskell (bad fit)
• Java (oh, common..)
Questions #3
IDE?
•Emacs
•VIM
Notes #1
•we use Go and Clojure for other systems
•do you want to ask “Why”?
•we are still on early production stage
•wait for new lessons coming soon
Ideas?
Questions?