Professional Documents
Culture Documents
Cassandra at Twitter
Cassandra at Twitter
Team
Chris Goffinet
Stu Hood
Ryan King
@stuhood
@rk
Melvin Wang
@alan
@padauk9
Measuring ourselves
#prostyle
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
Hardware Platform
CPU Core Utilization Memory bandwidth and consumption Machine cost RAID Filesystems and I/O Schedulers IOPS Network bandwidth Kernel
Hardware Platform
CPU Core Utilization Memory bandwidth and consumption Machine cost RAID Filesystems and I/O Schedulers IOPS Network bandwidth Kernel
Hardware Platform
Filesystem configurations
Ext4
XFS RAID
Hardware Platform
I/O Schedulers
Timeseries 50/50
Measure
Hardware Platform
I/O Schedulers 5050 - Reads
Scheduler cfq noop deadline anticipatory p90 73ms 47ms 75ms 76ms p99 210ms 167ms 233ms 214ms Average 11.72ms 9.12ms 12.72ms 12.37ms Max 4940ms 4132ms 3718ms 5120ms
Hardware Platform
I/O Schedulers 5050 - Writes
Scheduler cfq noop deadline anticipatory p90 2ms 2ms 2ms 2ms p99 2ms 2ms 2ms 2ms Average 2.02ms 2.06ms 2.13ms 2.03ms Max 5927ms 3475ms 3718ms 5119ms
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
Data Storage
How efficient is our on-disk storage? Could we do compression? Do we have CPU to trade? How do we push for better? Is it worth it?
Data Storage
Old Easy to Implement Checksumming Varint Encoding Delta Encoding Type Specic Compression Fixed Size Blocks New
Data Storage
Old Easy to Implement Checksumming Varint Encoding Delta Encoding Type Specic Compression Fixed Size Blocks X X X X X X New
Data Storage
Data Storage
Data Storage
7.03x
Data Storage
10,00o rows; 250M columns
Rows Current Format New Format Timeseries LongType column names CounterColumnType values 10000 10000 Columns 250M 250M Size on disk 16,716,432,189 2,375,027,696 bytes per column 66.8 9.5
Data Storage
compression
type specific
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
What are our issues? Compaction Performance? Caching? Too many disk seeks? Garbage Collection?
Compaction
Compaction
Multithread Compaction + Throttling Compact each bucket in parallel Throttle across all buckets Compaction running all the time CASSANDRA-2191 CASSANDRA-2156
Measure latency
p99 p999
No averages! Every customer has p99 and p999 targets we must hit 24x7 on-call rotation
Caching?
In-heap Off-heap
Pluggable cache
Memcache
Growth was requiring entire dataset in memory. Why? How big is the active dataset within 24hours? What happens when dataset outgrows memory? Could other storage solutions do better? What are we missing here?
or is it?
On-heap
Off-heap
Memcache
Out of process
On-heap
Off-heap
Memcache
Co-locate memcache on each node Routing + Cache replication Write through LRU Rolling restarts do not cause degraded performance states
Cassandra
Memcache
Cassandra
Memcache
Cassandra
Memcache
Cassandra
Memcache
99th percentile went before 200ms 800ms when data > memory 99th percentile now - 2.5ms
New observability stack Replaces Ganglia Collect metrics for graphing in real-time Scale based on machine count + defined metrics Heavy write throughput requirements SLA Target
1.3 million writes/second 112 billion writes a day 3.2 gigabit/s over the network 492GB of new data per hour 140MB/s writes across cluster 70MB/s reads across cluster
36,000 writes/second
36 nodes without RF (Replication Factor) Replication Factor = 3 30-35% cpu utilization FSync Commit Log every 10s
5.0e+08
value
1.5e+09
1.0e+09
max_chunk
5.0e+08
1000
2000
time
3000
4000
Slab Allocation Fixed sized chunks (2MB) Copy byte[] into slabs using CAS (Compare & Swap) Largely reduced fragmentation CASSANDRA-2252
30-60 seconds
Frequency of pause
Every hour
30-60 seconds
5 seconds
Frequency of pause
Every hour
3 days 10 hours
Pluggable Compaction
Make it easy to implement more interesting and intelligent compaction strategies SSTable Min/Max Timestamp
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
Operational Efficiency
Automated infrastructure burn-in process Rack awareness to handle switch failures Grow clusters per rack, not per node Lower Server RPC timeout (200ms to 1s)
Fail fast
Operational Efficiency
No swap and dedicated commit log Multiple hard drive vendors 300+ nodes in production Run on cheap commodity hardware Design for failure
Operational Efficiency
What failures do we see in production?
Bad memory that causes corruption Multiple disks dying on same hosts within hours Rack switch failures Memory allocation delays causing JVM to encounter higher latency GC collections (mlockall recommended) Stop the world pauses if traffic patterns change
Operational Efficiency
What failures do we see in production?
Network cards sometimes negotiating down to 100Mbit Machines randomly die and never come back Disks auto-ejecting themselves from the raid array
Operational Efficiency
Deploy Process
Driver Hudson Git
Cass
Operational Efficiency
Deploy Process
Disable Gossip on a node Check ring on all nodes to ensure Down state Drain Restart
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
Capacity Planning
hardware platform (kernel, hw data) on-disk serialization overhead cost of read/write (seeks, index overhead) query cost (cpu, memory usage) requirements from customers
Capacity Planning
Input Example
spec = { 'read_qps': 500, 'write_qps': 1000, 'replication_factor': 3, 'dataset_hot_percent': 0.05, 'latency_95': 350.0, 'latency_99': 250.0, 'read_growth_percentage': 0.1, 'write_growth_percentage': 0.1, ...... }
Capacity Planning
Output Example
90 days datasize: 14.49T page cache size: 962.89G number of disks: 68 disk capacity: 15.22T iops: 6800.00/s replication_factor: 3 servers: 51 servers (w/o replication): 17 read_ops: 2323 write_ops: 991066 servers: 57 servers (w/o replication): 19 read_ops: 2877 write_ops: 1143171
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
Developer Integration
Cassie
Light-weight Cassandra Client Cluster member auto discovery Uses Finagle (http://github.com/twitter/ finagle) Scala + Java support Open sourcing
Measuring ourselves
Hardware Platform Data Storage Latency and Throughput Operational Efficiency Capacity Planning Developer Integration Testing
Testing
Performance Framework
Performance Framework
Performance Framework
Payload Sizes: 5
Performance Framework
Summary
Understand your hardware and operating system Rigorously exercise your entire stack Capacity plan with math not guesswork Measure everything, then do it again Invest in your storage technology Automate Expect everything to fail