LLM-driven Data Engineering

Day 1
EcZachly Inc
Is data engineering doomed to
AI?
And what you can do about it!
EcZachly Inc
Here’s why data engineering is doomed
(according to shitty LinkedIn influencers)
EcZachly Inc

- LLMs are getting good at writing SQL

- LLMs increase productivity, reduce debug time, and reduce documentation
writing time
- LLMs can even write Spark code! Y’all saw the English SDK for Databricks
right? EcZachly Inc
- LLMs will guarantee amazing data quality!
LLMs and SQL
EcZachly Inc

I’m sure you’ve seen the

SQL is an important part of the data engineering equation

- LLMs are good but not great at SQL (we will see this in the lab today)
EcZachly Inc
I’m going to show you a chart soon
EcZachly Inc

- Red means, LLMs are going to have a significant impact, soon

- Yellow means, LLMs aren’t quite there yet but could disrupt in the future
- Green means, LLMs are far from accomplishing this

EcZachly Inc
EcZachly Inc

EcZachly Inc
Answering business questions (red)
EcZachly Inc

LangChain hooks into databases

directly and can easily make sense
of data warehouses!

This disruption is imminent! EcZachly Inc

Fixing Broken Pipelines (red)
EcZachly Inc

LLMs + Agents will be able to

significantly reduce the oncall burden.

- LLMs that process stack traces /

quality failures can recommend
possible remedies to unblock
(maybe even via Slack)

The amount of data engineering hours

EcZachly Inc
that are recovered from this is
something to be celebrated!

Tricky failures will still need manual

troubleshooting though
Writing analytical queries (red)
EcZachly Inc

- This will be disrupted 100% but complexity

matters here a lot.
- SQL practitioners probably won’t feel much
difference here
- Self-serve analytics will catch fire here though
This is something to be celebrated!
EcZachly Inc
Data engineers won’t have to hand hold
stakeholders through their analytical questions
nearly as much!
Writing pipeline code in:
SQL, Spark, or Flink (orange)
EcZachly Inc

LLMs can oftentimes give us a

good boilerplate to start with, but
debugging it is often not worth it!

We are still pretty far from

EcZachly Inc
prompts generating us optimized
and correct data pipelines!
Why ChatGPT sucks for SQL!
EcZachly Inc

LLMs are like having a good junior engineer

who can write SQL for you but you have to
check their work

The chat.openai.com interface is

NON-DETERMINISTIC!
EcZachly Inc
The same prompt will output different things
depending on when you input it!

REMEMBER FROM CLASS, IF THINGS ARE NOT

IDEMPOTENT, THEY ARE NOT TO BE TRUST
Debugging
EcZachly Inc

EcZachly Inc
Writing data documentation (orange)
EcZachly Inc

If you’ve done the upfront work of all the

conceptual and physical data modeling,
ChatGPT can give you beautiful
summaries and example queries. (We’ll
see this in the lab today)
ChatGPT will miss most business EcZachly Inc
context unless directly given it and if
you’re directly giving ChatGPT the
context, you might as well write the
documentation yourself at that point
Sprint Planning (orange)
EcZachly Inc

LLMs should be able to give us a

good idea about sprint planning.

The soft skills needed to prioritize EcZachly Inc

and push back are sadly not
present in ChatGPT
Dashboarding (orange)
EcZachly Inc

Presenting data is actually pretty

easy. Usually simple GROUP BY
queries does the job

Knowing you customer and what

EcZachly Inc
data to display and when is the
hard part that ChatGPT will
continue to suck at
Physical Data Modeling (orange)
EcZachly Inc

Optimizing logical/conceptual
data models to a specific
architecture is something
ChatGPT will excel at.
Having all the best practices in EcZachly Inc
place and understand the interplay
between data table, pipeline, etc is
hard for ChatGPT to understand
Writing tests for you pipelines (orange)
EcZachly Inc

Generating fake data will be a big

pain that LLMs can give us
already.

EcZachly Inc
Knowing if we have adequate test
coverage is something ChatGPT
will continue to be bad at
Automated Data Quality Checks (orange)
EcZachly Inc

Suggesting automated checks is

something LLMs can do already

Having the business context to EcZachly Inc

know what checks are valuable is
something LLMs will have a hard
time with
Building data processing frameworks
(green)
EcZachly Inc

Once you hit a sufficient level of

complexity, LLMs fall apart.

Imagine asking ChatGPT,

“Give me the code for the next EcZachly Inc
generation of Spark”
Creating data best practices (green)
EcZachly Inc

Data best practices are most

often learned from experience.
Experience that needs to
permeate through the
organization. EcZachly Inc

ChatGPT will catch up 2-3 years

late!
Conceptual / Logical Data Modeling (green)
EcZachly Inc

Data Modeling is a very soft skills

focused task.

Understanding what data is

needed, where it is, and how to get EcZachly Inc
it is a human-oriented task that
ChatGPT will continue to struggle
with
What’s the lab going to be about today?
EcZachly Inc

- When you should and shouldn’t use LLMs

- What types of tasks are LLMs better suited for
- How can you get the most out of each prompt?

EcZachly Inc
When should you use LLMs?
EcZachly Inc

- Debugging specific problems (similar to StackOverflow or Google)

- Getting your starter DAG boilerplate
- Boilerplate documentation
- Solving a very specific part of a query
EcZachly Inc
Generating Documentation
EcZachly Inc

- LLMs are actually decent at generating text and documentation

- Business context is often lacking though!
- Make sure to specify to use Markdown in the prompt
- The more context you give, the better the documentation
EcZachly Inc
How to use the OpenAI API
EcZachly Inc

- Install the openai Python library

- Get an OpenAI API key (REMEMBER ITS PAY PER USE)

EcZachly Inc
How the prompts work in Python API
EcZachly Inc

- You have multiple roles you can play

- System
- Assistant
- User

EcZachly Inc
The System Role
EcZachly Inc

Think of this as the “contextual” clues that guide the prompt the right way

EcZachly Inc
The Assistant Role
EcZachly Inc

This is for when you want to have a multiple prompt/response conversation

with ChatGPT

EcZachly Inc
The User Role
EcZachly Inc

This is the prompt that is used for the specific task at hand

EcZachly Inc
API Configuration
EcZachly Inc

Temperature - Scales from 0 to 2. Closer to 0 the more deterministic the

output.
Max Tokens - you can set this as high as 4000. One token is about 4 characters
in English
EcZachly Inc
Today’s Lab
EcZachly Inc

- We’ll be interacting with GPT-4 in today’s lab

- We’ll generate an Airflow DAG
- A SQL query
- And documentation for our data sets

EcZachly Inc