Streaming

Jamie Grier | @jgrier

1
Agenda

• Goals of Lyft’s Streaming Platform

• Streaming Platform Overview

• Why Flink

• Why Kafka

• Open problems

2
Goals of Lyft’s Streaming Platform

• Make it easy to build real-time, event-driven, stateful,

microservices

• Solve the hard parts of stream processing ONCE for the entire
company

• Be a force multiplier for other teams within Lyft

3
Streaming Platform Overview
Stream Compute
Pub/Sub Streaming Pub/Sub
Service One

Streaming
Service Two

Streaming
Service Three

Stream / Schema Deployment Metrics &

Alerts Logging
Registry Tooling Dashboards

Amazon Salt
Amazon S3 Wavefront Docker
EC2 (Conifg / Orca) 4
Lyft Streaming Platform - Streaming Compute Criteria

API Considerations: Operational Considerations

● Functional / Fluent API
● Flexible Windowing API
● Event Time Support
● Apache Beam Support
● Stream SQL
● Powerful Direct API
● Late Data Handling
● Stateful Computation and Exactly-
once Processing Semantics
● Robust State Management
● Data Reprocessing (backfill)
● Asynchronous Checkpoints
● Back-pressure
● High throughput and low-latency
● Deployment Architecture

The contenders: Apache Flink, Apache Spark Streaming, Apache Kafka Streams
5
Why Flink? API Considerations
• Functional / Fluent API
• Flexible Windowing API
• Event Time Support
• Apache Beam Support
• Stream SQL
• Powerful Direct API
• Late Data Handling

6
Why Flink? Operational Considerations

• Stateful Computation and Exactly-once Processing Semantics

• Robust State Management
• Stateful Data Reprocessing (backfill)
• Asynchronous Checkpoints
• Back-pressure
• High throughput and low-latency
• Deployment Architecture

7
Lyft Streaming Platform - Pub/Sub Criteria

Semantics / Features Operational Considerations

● Write Latency
● Durability
● Read Latency
● Consumer Fanout
● Project Maturity
● Transactions / Idempotent Writes ● Vendor Support
● Per-Key Ordering Guarantees
● Long-Term Data Storage
● Auto-Scaling

The contenders: Apache Kafka, Amazon Kinesis, Pravega

8
Why Kafka?
Pros
• Durability & Write Latency
• Read Latency & Consumer Fanout
• Transactions & Idempotent Writes
• Operational Concerns & Vendor Support
Cons
• No ordering by key, only partition
• Long term data storage still an issue
• Auto-Scaling still an issue

9
Open Problems

• Rescaling Kafka while preserving per-key ordering

• Efficient Dynamic Computations over streams

• Long term storage for events: real-time and historical reads

• Zero Downtime deployments for streaming services

10
Rescaling Kafka

• Rescaling Kafka while preserving per-key ordering

• Kafka only provides partition ordering guarantees!

• We want per-key ordering guarantees

• Guarantees should hold across re-partitioning events

• Basic approach: Read old partitions completely before reading

new

• Achieve this using something akin to Flink’s checkpoint 11

Rescaling Kafka while preserving per-key ordering

12
Rescaling Kafka while preserving per-key ordering

13
Efficient Dynamic Computation Over Streams

• Enable many users to dynamically submit small streaming

computations

• Share bandwidth amongst multiple computations

• Share computed sub-results amongst multiple computations

• Correctly handle bootstrapping of computations which

depend on historical data

• Basic approach: Map any computation into a fixed/general

14
Efficient Dynamic Computations over streams

15
Efficient Dynamic Computations over streams

16
Long term storage for events: Real-time and historical reads

17
Zero Downtime deployments for streaming services

18
Summary

• Lyft is building a next generation streaming platform based

on Apache Flink and Apache Kafka

• Stateful stream processing is not a “solved problem”

• There are many hard / open problems left to solve

• If these sort of problems interest you please come join us!

We’re Hiring!
19
Thank you!
Jamie Grier

20