Welcome!

Introduction to
Data Engineering
With Delta
Your Hosts for Today

Joel Roland Noble Raveendran

SA Manager, ANZ Solutions Architect
Agenda

▪ Overview of Data Engineering powered by Delta Lake

▪ Demo
▪ Q&A
Housekeeping

▪ If you have questions during the session please post in the Chat
Window

▪ We will have a number of Polls during the event - they will pop up so
please respond when they do
 The Data Engineer’s Journey...

Events
Stream

Stream Unified View AI & Reporting

Validation

Updates & Merge get complex

Batch Batch
Table with data lake
Table
(Data gets compacted
(Data gets written
every hour) Update & Merge
continuously)
Reprocessing
 The Data Engineer’s Journey...

Events

d ?
ifie
Stream

p l
be sim AI & Reporting

his
Stream Validation Unified View

n t
Ca
Batch Batch Updates & Merge get complex
Table with data lake
Table
(Data gets compacted
(Data gets written
every hour) Update & Merge
continuously)
Reprocessing
 A Data Engineer’s Dream...

Process data continuously and incrementally as new data arrives in a

cost-efficient way without having to choose between batch or
streaming

Kinesis

CSV, AI & Reporting

JSON, TXT…
Data Lake
 What’s missing?
Kinesis
CSV,
?
JSON, TXT… AI & Reporting
Data Lake

• Ability to read consistent data while data is being written

• Ability to read incrementally from a large table with good throughput

• Ability to rollback in case of bad writes

• Ability to replay historical data along new data that arrived

• Ability to handle late arriving data without having to delay downstream processing
 So… What is the answer?

Delta
+ = Architecture
STRUCTURED
STREAMING

1. Unify batch & streaming with a continuous data flow model

2. Infinite retention to replay/reprocess historical events as needed
3. Independent, elastic compute and storage to scale while balancing
costs
Let’s try it instead with
Well…what is
?
Data reliability challenges with data lakes
Failed production jobs leave data in corrupt
✗ state requiring tedious recovery

Lack of schema enforcement creates

inconsistent and low quality data

Lack of consistency makes it almost impossible to

mix appends and reads, batch and streaming
Performance challenges with data lakes
Too many small or very big files - more time opening &
closing files rather than reading contents (worse with
streaming).

Partitioning aka “poor man’s indexing”- breaks down if

you picked the wrong fields or when data has many
dimensions, high cardinality columns.

No caching - cloud storage throughput is low (cloud object

storage is 20-50MB/s/core vs 300MB/s/core for local
SSDs).
A New Standard for Building Data Lakes

Open Format Based on Parquet

With Transactions

Apache Spark API’s

Delta Lake: makes data ready for analytics

Data Science & ML

Reliability

Performance
• Recommendation Engines
• Risk, Fraud Detection
• IoT & Predictive Maintenance
• Genomics & DNA Sequencing
 Delta Lake ensures data reliability

Batch
Parquet Files
Streaming
High Quality & Reliable Data
always ready for analytics
Updates/Deletes Transactional
Log

Key Features ● ACID Transactions ● Unified Batch & Streaming

● Schema Enforcement ● Time Travel/Data Snapshots
 Delta Lake optimizes performance
Databricks
optimized engine

Highly Performant
queries at scale

Parquet Files

Transactional
Log

Key Features ● Indexing ● Data skipping

● Compaction ● Caching
Now, let’s try it with
 *Data Quality Levels *
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting
Quality

Delta Lake allows you to incrementally improve the

quality of your data until it is ready for consumption.
What does this remind you of ?
 Data Lifecycle
Bronz Silve Gold
e r Streamin
Kinesis
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting

Quality
 Data Lifecycle
OLTP
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting

Quality
 Data Lifecycle
OLTP Staging
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting

Quality
 Data Lifecycle
OLTP Staging DW/OLAP
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting

Quality
Data Lifecycle 🡪 Delta Lake Lifecycle
 *Data Quality Levels *
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake Ingestion Aggregates
Augmented
AI &
Reporting
Quality

Delta Lake allows you to incrementally improve the

quality of your data until it is ready for consumption.
 OLTP

Bronz Silve Gold

Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…

Data Lake
Raw Filtered, Cleaned Business-level
Ingestion Augmented Aggregates
AI &
Reporting
• Dumping ground for raw data
• Often with long retention (years)
• Avoid error-prone parsing
 Staging

Bronz Silve Gold

Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…

Data Lake
Raw Filtered, Cleaned Business-level
Ingestion Augmented Aggregates
AI &
Reporting
Intermediate data with some cleanup applied
Query-able for easy debugging!
 DW/OLAP

Bronz Silve Gold

Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting
Clean data, ready for consumption
Read with Spark and other SQL Query Engines
 Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting
Streams move data through the Delta Lake
• Low-latency or manually triggered
• Eliminates management of schedules and jobs
 OVERWRITE

MERGE
INSERT DELETE
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
UPDATE AI &
Reporting
Delta Lake also supports batch jobs
and standard DML
• Retention • GDPR
• Corrections • UPSERTS
 DELETE DELETE
Bronz Silve Gold
Kinesis e r Streamin
g
Analytics
CSV,
JSON, TXT…
Raw Filtered, Cleaned Business-level
Data Lake
Ingestion Augmented Aggregates
AI &
Reporting

Easy to recompute when business logic changes:

• Clear tables
• Restart streams
 Delta
Architecture
 Connecting the dots...
Kinesis
CSV,
? AI & Reporting
JSON, TXT…
Data Lake
 Connecting the dots...
Kinesis
CSV,
? AI & Reporting
JSON, TXT…
Data Lake

• Ability to read consistent data while data is Snapshot isolation between writers and
being written readers
 Connecting the dots...
Kinesis
CSV,
? AI & Reporting
JSON, TXT…
Data Lake

• Ability to read consistent data while data is Snapshot isolation between writers and
being written readers

• Ability to read incrementally from a large Optimized file source with scalable
table with good throughput metadata handling
 Connecting the dots...
Kinesis
CSV,
? AI & Reporting
JSON, TXT…
Data Lake

• Ability to read consistent data while data is Snapshot isolation between writers and
being written readers

• Ability to read incrementally from a large Optimized file source with scalable
table with good throughput metadata handling

• Ability to rollback in case of bad writes Time travel

 Connecting the dots...
Kinesis
CSV,
? AI & Reporting
JSON, TXT…
Data Lake

• Ability to read consistent data while data is Snapshot isolation between writers and
being written readers

• Ability to read incrementally from a large Optimized file source with scalable
table with good throughput metadata handling

• Ability to rollback in case of bad writes Time travel

• Ability to replay historical data along new Stream the backfilled historical data
data that arrived through the same pipeline
 Connecting the dots...
Kinesis
CSV,
? AI & Reporting
JSON, TXT…
Data Lake
• Ability to read consistent data while data is Snapshot isolation between writers and
being written readers

• Ability to read incrementally from a large Optimized file source with scalable
table with good throughput metadata handling

• Ability to rollback in case of bad writes Time travel

• Ability to replay historical data along new Stream the backfilled historical data
data that arrived through the same pipeline

• Ability to handle late arriving data without Stream any late arriving data added to the
having to delay downstream processing table as they get added
 Connecting the dots...
Kinesis
CSV,
JSON, TXT… AI & Reporting
Data Lake
• Ability to read consistent data while data is Snapshot isolation between writers and
being written readers

• Ability to read incrementally from a large Optimized file source with scalable
table with good throughput metadata handling

• Ability to rollback in case of bad writes Time travel

• Ability to replay historical data along new Stream the backfilled historical data
data that arrived through the same pipeline

• Ability to handle late arriving data without Stream any late arriving data added to the
having to delay downstream processing table as they get added
Get Started with Delta using Spark APIs
Instead of parquet... … simply say delta

CREATE TABLE ... CREATE TABLE ...

USING parquet USING delta
... …

dataframe dataframe
.write .write
.format("parquet") .format("delta")
.save("/data") .save("/data")
Using Delta with your Existing Parquet
Tables
Step 1: Convert Parquet to Delta Tables

CONVERT TO DELTA parquet.`path/to/table` [NO STATISTICS]

[PARTITIONED BY (col_name1 col_type1, col_name2 col_type2, ...)]

Step 2: Optimize Layout for Fast Queries

OPTIMIZE events
WHERE date >= current_timestamp() - INTERVAL 1 day
ZORDER BY (eventType)
Upsert/Merge: Fine-grained Updates
MERGE INTO customers -- Delta table
USING updates
ON customers.customerId = updates.customerId
WHEN MATCHED THEN
UPDATE SET address = updates.address
WHEN NOT MATCHED THEN
INSERT (customerId, address) VALUES (updates.customerId, updates.address)
Time Travel
Reproduce experiments & reports Rollback accidental bad writes

SELECT count(*) FROM events INSERT INTO my_table

TIMESTAMP AS OF timestamp SELECT * FROM my_table TIMESTAMP AS OF

date_sub(current_date(), 1)

SELECT count(*) FROM events VERSION

AS OF version

spark.read.format("delta").option("timestampAsOf", timestamp_string).load("/events/")
 Databricks Ingest: Auto Loader
Load new data easily and efficiently as it arrives in cloud storage

Before After
Notification Message Auto
Service Queue Loader

Stream

Batch

Delayed ● Pipe data from cloud storage into Delta Lake as

schedule
it arrives
External Trigger
● “Set and forget” model eliminates complex setup
Airflow file sensor

Gets too complicated for multiple jobs

Launch Blog Post
Databricks Ingest: Data Ingestion Network
All Your Application, Database, and File Storage Data in your Delta Lake
Delta Lake Connectors
Standardize your big data storage with an open format accessible from various tools

Amazon Redshift

Amazon Athena
Databricks Notebooks are a powerful developer environment
to enable Dashboards, Workflows, and Jobs.

Notebook Dashboards
Databricks Notebooks Create interactive notebooks based on
Collaborative, multi-language, Notebooks with just a few clicks.
enterprise-ready developer environment.

Notebook Workflows
Orchestrate multi-stage pipelines based on
notebook dependencies.

Notebook Jobs
Schedule, orchestrate, and monitor
execution of notebooks.
 Notebook Workflows enable orchestration of
multi-stage pipelines based on notebook dependencies.

Workflow Definition
APIs allow flexible definition of notebook
dependencies, including parallel
execution of notebooks.

Databricks Job Scheduler

Workflow Execution
The Databricks Job scheduler
manages the execution of
Notebook workflows.
 Notebook Jobs enable scheduling and monitoring
of Notebooks.

Turn Notebooks into Jobs

Any Databricks Notebook or Notebook
Workflow can easily be turned into a job.

Schedule Jobs
Jobs can be configured to execute on a
schedule (e.g. daily, hourly).

Orchestrate Jobs
In addition to Notebook Workflows, Jobs
can also be orchestrated using third-party
tools like Airflow.
Users of Delta Lake
Improved reliability:
Petabyte-scale jobs

10x lower compute:

640 instances to 64!

Faster iterations:
Multiple wks to 5 min
deploys!
● 2 Regions Uses Cases
● 10 Workspaces Improved performance:
● Finance
● 100+ Users Queries run faster
● 50+ Scheduled ● Assortment
Jobs
>1 hr → < 6 sec
● Fresh Sales
● 1,000+ Notebooks
Tool
● Scores of ML
● Fraud Engine
Easier transactional
models
● Personalization updates:
No downtime or
consistency issues!

Simple CDC:
Easy with MERGE
 Data consistency and integrity:
not available before

Increased data quality:

name match accuracy up from
80% to 95%

Faster data loads:

24 hours → 20 mins

Databricks Cluster
Demo
Q&A
Your feedback is appreciated

First 20 responses will receive a $10

UberEats voucher
TWO LIVE SESSIONS:
Australia & NZ: 9.00am - 11.30am AEST
Singapore & India: 1.00pm - 3.30pm SGT | 10.30am - 1.00pm IST

Hear the data innovation stories of Atlassian, Coles Group, Grab

Tabcorp, and more.
Register now: https://databricks.com/p/event/data-ai-tour
 June 22-26 | Organized by

THE FREE VIRTUAL

EVENT FOR DATA TEAMS
● Extended to 5 days with over 200 sessions
● 4x the pre-conference training
● Keynotes by visionaries and thought
leaders
https://databricks.com/sparkaisummit/north-america-2020