Eventbrite:

Our Move from Colo to EC2

Or…
How I Learned To Stop Worrying
and Love The Cloud
Have you seen one of these
before?
It’s kinda like one of these....
Strangely related to this?
 Or this
“dumb” terminal?
Where might you have seen these all in the
same place?
1996 playboy.com
Okay, no, we didn’t run the
web servers on a Token Ring
Network
We ran them on these instead
Web and DB layer
 So...What’s the Point?
• Specific technologies work for a specific solution
until your needs change or a more
efficient/economical/powerful technology occurs.
• Sometimes, it’s better to have a slow but stable
technology in place, when you are messing around
with riskier ones, like an Internet-based business in
1996.
• Playboy’s Internet team was one of the first “lean
startups”.
Fast-forward to Digg.com
March 2006
 March, 2006
• I joined Digg.com
• 1,100 requests a second at peak
• Roughly 1 million Unique Visitors per month
• 5 cabinets of servers in an Equinix Datacenter
• A pair of misconfigured Netscaler load balancers
• 1 really good systems engineer and 1 really good
DBA
• No cloud
 Operational Challenges
• “What? We’re down again?”
• “The site is slow.” “Works for me.”
• Buying servers takes time and is expensive.
• Colocation and bandwidth are expensive.
• We had to buy several more cabinets worth of
servers in anticipation of the release of Digg 3.0 in
July.
• Early load testing told us we needed even more.
The solution: More servers in the
face of uncertainty
 Lather, Rinse, Repeat…
fast-forward to March 2009
 40 cabinets
and growing to over 60,
plus 13 in another DC for DR
More Datacenter Pr0n
Digg Production Diagram
(pre-v4)
• 2 data centers
• 840 servers
• But very good automation, if
you have to roll out a lot of
hardware fast, you figure out
how to automate
• We should’ve built our own
cloud service. We were that
good at rolling out cabinets of
servers.
 What if EC2 had existed in Oct. 2005?
• Managed hosting was expensive and a on-
demand solution would have been great as an
alternative to colocation, which has multi-year
contracts generally.
• Digg could have scaled up as traffic did and
then down if it became less popular.
• We would have been forced by EC2 limitation
to get smarter with our application design.
 Move from hosted servers to EC2
• Initially, much trepidation at the switch.
• Team moving to using Puppet with single
source of truth database, code-name iman, for
managing the clusters.
• Master AMI very important first step.
• Moving from Debian to Ubuntu
• DNS/DHCP a challenge (email/change of private
IP address)
 The Cloud
• Although the term inspires fear and loathing, it’s
turning out to be highly beneficial.
• EC2 makes scaling up and scaling down very simple.
• To fully leverage Agile Development, you need Agile
Operations. Cloud services help with that.
• EC2’s limitations can inspire creative and efficient
solutions; sometimes it helps to have limitations.
• Throwing instances, instead of hardware, at a
problem might not work (db connections).
 Timesharing
• Cloud computing is timesharing updated.
• Timesharing invented to allow computing power
without the hardware investment.
• Computers got cheap, timesharing goes away
(Security also a factor).
• Large colo farms expensive, cloud computing
invented to lower entry barrier (Security still a
problem).
 Why Cloud-Based Cluster?
• Not for cost reasons as you grow: We could do our
current server configuration cheaper in a colo.
• Flexibility: this is the main reason. We want special
environments for special ticketing events, no problem.
We use Puppet to spin up some more instances and use
haproxy to separate the traffic.
• Easier to experiment with new configs, less worry about
hardware. Limitations help focus on fixing inefficiencies.
• Scaling the hardware is someone else’s problem.
 Some Tips on Building on EC2
• If you are using EBS-backed instances for your
datastores, use RAID0, instead of 10.
• If you need to send email, and you want it
delivered, use a 3rd party service or lease a
cabinet in a colo.
• Defense in depth, redundancy in depth,
backups in depth.
• If the instance is important, use EBS.