Professional Documents
Culture Documents
The Ebay Architecture: Amazon Friday Learning Series
The Ebay Architecture: Amazon Friday Learning Series
Striking a balance between site stability, feature velocity, performance, and cost
• eBay manages …
– Over 212,000,000 registered users
– Over 1 Billion photos
An SUV is
A sporting soldsells
good every 5 minutes
every 2 seconds
– eBay users worldwide trade more than $1812
worth of goods every second
– eBay averages over 1 billion page views per
day
– At any given time, there are approximately
100 million listings on the site
– eBay stores over 2 Petabytes of data – over
200 times the size of the Library of Congress!
– The eBay platform handles 3 billion API calls
per month Over ½ Million pounds of
Kimchi are sold every year!
• In a dynamic environment
– 300+ features per quarter
– We roll 100,000+ lines of code every two weeks
June Q3 Growth
1999 2006
Outbound Emails 1M 35 M 35x
43 mins/day 50 sec/day
105
Million
Listings
212
Million
Users
Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3
Feature
212M Users
• Our site is our product. We change it incrementally through implementing new features.
• Very predictable development process – trains leave on-time at regular intervals (weekly).
• Parallel development process with significant output -- 100,000 LOC per release.
• Always on – over 99.94% available.
Availability
Reliability
Massive Scalability
Enable seamless growth
Security
• Virtualize Components
– Reduce physical dependencies.
– Improve deployment flexibility.
212M
Registered Users
V3 V4
eBay architecture versions
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
10
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
11
However …
By November 1999, the database servers approached their limits of physical growth.
12
13
BATCH JOBS
Bear
Tran
CATY 5 CATY 6 CATY 7 CATY 8
Scratch
CATY 9 CATY 10 CATY 11 CATY 12
December,1999
November, 2002
SUN
SUN
14
A3500
© 2006 eBay Inc.
Now that we have the Database taken care of….
• Application Server
– Monolithic 2-tier Architecture
– 3.3 Million Line C++ ISAPI DLL (150MB binary)
– Hundreds of developers, all working on the same code
– Hitting compiler limits on number of methods per class (!!)
15
16
18
• Rationale
– Partitions data by different scaling / usage characteristics
– Supports functional decoupling and isolation
19
• Rationale
– Horizontal scaling of transactional load.
– Segment business impact on database outage.
20
Application Servers
21
22
• Rationale
– Avoid deadlocks
– Avoid coupling availability
– Update concurrency
– Seamless handling of splits
23
• Minimize dependencies
– Between applications
– Between functional areas
– From applications to data tier resources
25
26
XSL
Presentation Tier
Command (View)
XML Model
AO/AOF (View) Building Logic
27
28
• Restrict inter-dependencies
• Applications depend on Domains, not on other Applications
• No dependencies among shared Domains
Applications
UserDomain SellingDomain BuyingDomain BillingDomain SearchDomain
29
LookupDomain
© 2006 eBay Inc.
Scaling the Application Tier – Functional Segmentation
CGI0 CGI5
Load Balancing
… …
IIS WebServers
AppServers
Load Balancing
AS AS AS AS AS AS
Load Balancing
…
30 User Acct Caty1 Caty20+
© 2006 eBay Inc.
Scaling the Application Tier – Platform Decoupling
Transaction
Platform
SOA
EDA
EDA
31
33
34
Item
Feeding System Search Engine
Data V3 Search
Voyager Frontend
Voyager
Voyager
Node
Node
Node
V3 Search
Frontend
Scrubber
Item Voyager
Voyager
Data Voyager
Node V3 Search
Node Search
Node Search Frontend
Search
Intelligence
Intelligence
Intelligence
V3 Search
Publishing
M Voyager Frontend
Voyager
e Voyager
Node
Item Search Node
PARd s Node
Data Database Aggregator V3 Search
s
a Frontend
Recovery
g
Search EOA Voyager
I Voyager
Voyager
Node V3 Search
Publishing n Node
Node Frontend
g
Search PARd
Item
Database B V3 Search
Data
u Voyager Frontend
Recovery Voyager
s Voyager
Node
Node
Node
V3 Search
Frontend
V3 Search
Voyager
Voyager Frontend
Indexer Voyager
Voyager
Node
Item Node
Node
Data
35
• Demanding Requirements
– Entire site rolled every 2 weeks
– All deployments require staged rollout with immediate rollback if necessary.
– More than 100 WAR configurations.
– Dependencies exist between pools during some deployment operations.
– More than 15,000 instances across eight physical data centers.
• Rollout Plan
– Custom application that works from dependencies provided by projects.
– Creates transitive closure of dependencies.
– Generates rollout plan for Turbo Roller.
• Automated Rollout Tool (“Turbo Roller”)
– Manages full deployment cycle onto all application servers.
– Executes rollout plan.
– Built in checkpoints during rollout, including approvals.
– Optimized rollback, including full rollback of dependent pools.
37
38
Availability
Enabling seamless growth
Reliability
Massive Scalability • Massive Database and Code Scalability
Security
39