APPLYING THE AGILE MANIFESTO TO

YOUR DATA WAREHOUSING PROGRAM

2 Why is an Agile approach beneficial for data warehousing?
3 The 12 principles of Agile, adapted for data warehousing
3 Satisfy the customer
3 Welcome change
4 Deliver frequently
4 Work together
5 Support your team
5 Interact face-to-face
6 Measure progress
6 Promote sustainable development
7 Conduct frequent design reviews
7 Keep it simple
8 Enable self-organizing teams
8 Reflect on effectiveness
9 Conclusion
10 The 12 principles of Agile adapted for data warehousing
11 About Snowflake
 CHAMPION GUIDES
WHY IS AN AGILE APPROACH
BENEFICIAL FOR DATA WAREHOUSING? Agile development:
A refresher
Despite the clear benefits of data warehouse Simply copying Agile best practices from the The 12 Principles of Agile were first put forth in a
software world would not fit well with the realities manifesto published by a consortium of software
programs, their value is often questioned by
and complications of data warehousing; they need developers in 2001. They sought to codify a more
a perception of long delivery timelines and nimble, people-focused approach to software
to be modified. Two-week sprints, for example, are
a lack of clear and timely business value. In development. Agile quickly became the industry
considered the ideal in software development, but
fact, nearly half (48%) of the respondents in a they are not always optimal with data warehouse
standard, and it is often described more like a
TDWI survey reported that personnel in their mindset or even a belief system than merely a
projects (though occasionally doable). As a result, I’ve
organizations spend at least 61% of their time set of tactics.
adapted Agile principles over years of work across
finding and preparing data, and only 28% said multiple organizations, and I have seen firsthand just
Agile development is characterized by the following:
that their users and analysts can access and how powerful this approach can be.
● Work is done in iterative pieces that are refined
analyze new data, including external data, By adopting an Agile approach and modifying it for continuously based on feedback from
without close IT support.¹ This is no longer our data warehousing needs, we can change the business stakeholders.
acceptable in the ever changing and evolving perceptions of data warehousing. We can learn to ● Work is done in a predetermined timeframe, often
world of data and analytics. The time to value deliver value more quickly by being adaptable and known as a sprint. In software development, sprints
flexible. are usually two weeks.
must be decreased.
In the following pages, you’ll find the foundational 12 ● Each sprint is made up of the smallest possible
The question driving many forward-thinking data deliverable or set of deliverables for that timeframe,
Principles of Agile, along with my recommendations
warehousing and BI departments is: How can we sometimes referred to as the minimum viable
for adapting them to data warehousing. I hope you’ll
deliver results faster, with closer integration between product (or MVP).
find the approach as transformational as I have.
data teams and their business stakeholders? A key
to solving this problem clicked into place, for me,
According to MIT Sloan School of Management,
over a decade ago when a CIO handed me a book Sincerely, Agile development is in contrast to a “traditional,
on Agile project management, written for software Kent Graziano linear ‘waterfall’ development model, in which entire
teams. Receiving that book started me on a journey Chief Technical Evangelist & Strategic Advisor, projects are preplanned, then fully built out before
of transforming data warehousing teams to be more Snowflake they are tested. Agile’s approach offers iterative
nimble and iterative, through a hybrid approach flexibility, with small parts of projects being built
that borrows from Agile concepts and applies the and tested simultaneously.”
techniques to data warehousing.

1
TDWI: Reducing Inefficiency and Increasing the Value of Analytics and Business Intelligence. 2018.
2

THE 12 PRINCIPLES OF AGILE,
ADAPTED FOR DATA WAREHOUSING
1. SATISFY THE CUSTOMER
Our highest priority is
to satisfy the customer
through early and continuous
delivery of valuable software.
Data warehouse adaptation:
Identify a proxy for the end customer, if
needed. That person or proxy must be able
to define what constitutes success.
2. WELCOME CHANGE
Welcome changing requirements,
• Make sure you have a flexible data platform that
even late in development. Agile
processes harness change for the
customer’s competitive advantage.
Data warehouse adaptation:
Establish clear user stories for every
requirement. To stay nimble, leverage
code generators, Agile data engineering
techniques, and a data platform that can
grow and change with you.
CHAMPION GUIDES
evolve or change, while building the foundation
for long-term storage and integration of data
from multiple systems, enabling greater flexibility
and resilience.
allows you to adapt quickly. Earlier in my career, I
was on a project where we ran out of disk space.
The recommended solution to our team was to
delete data. Clearly, this reflects a platform failure
and not a failure of the team. Using a platform that
can scale up and down with your needs will help
your team be more agile.

This principle may sound straightforward, but in data

warehousing, the question of who the customer
is and what constitutes “valuable” can be tricky to
define. For example, ETL code isn’t visible to the end
business user, though business reports often depend
on it. Is the end business user your customer, even
if that user can’t evaluate the code? Does ETL code
constitute a valuable piece of software?
Putting aside semantics, the “customer” in this case
might be a proxy, rather than the end user, and
that is acceptable as long as the proxy can define
what constitutes success. In the world of data
warehousing, valuable software may or may not
be visible to the end user. BI reports, dashboard
interfaces, or working ETL code all count as valuable,
long as they’re helping answer meaningful business
questions with data.
To react quickly to change, data teams must be as
nimble as possible. These four tips will help you
transition teams to a more adaptable, Agile approach:
• Establish clear user stories for every incoming
requirement. Good user stories outline the
following: Who is the actor, what does the actor
need, and what is the business goal (the “why”)?
• Use code generators. Generated code is easier to
revise and less error-prone.
• Adopt Agile data engineering techniques. Certain
data modeling techniques and architectural
approaches enable more Agile development.
Data Vault, for example, was designed to reduce
reengineering and refactoring as requirements

3

3. DELIVER FREQUENTLY
Deliver working software
frequently, from a couple of
weeks to a couple of months,
with a preference to the shorter timescale.
Data warehouse adaptation:
Focus on shortening your delivery iterations
from the length they are now.
Two-week sprints do not generally make sense for
the data warehouse world, but three-month cycles
are usually an improvement from the norm. Figure
out what works for your teams and business needs,
and proceed from there.
To shorten your delivery cycle, you need to enforce
good scope control and have a prioritized backlog.
Remember, a backlog isn’t a bad thing. It’s simply
your requirements list, prioritized by the business
owner. When new requirements arise, they initiate a
new user story. The business manager (also known as
the product owner) then determines where that new
user story falls in your priority list. That is why it is
important for every user story to include the business
reason for the requirement.
CHAMPION GUIDES
Finally, to maintain a shorter development cycle, be
sure to tackle one subject area at a time. Keeping
longer meets business objectives. Enforcing a tighter
integration between business and development
units of work small will help team members go home teams helps avoid failure altogether, and makes
and feel they’ve accomplished their goal for that day. warning signs of potential failure visible earlier in the
process.
Even if you aren’t delivering a product every day, you
4. WORK TOGETHER can deliver progress updates every day. Daily updates
to business stakeholders build confidence that you
are building what they need.
Business people and developers
must work together daily
throughout the project.
Data warehouse adaptation:
Ask business owners to join your daily
meetings to ensure continued buy-in and
minimize surprises.
Not involving business stakeholders is one of the top
reasons data warehouse projects fail. Failure is, after
all, a business designation for a deliverable that no
4

For example, at one large enterprise, we built a
global “war room” for an important data warehouse
decommission and replacement project. We had to
reengineer and move a mission critical data mart in
under three months. We had two senior directors
work daily with the data warehouse team for six
weeks. That tight integration was the only way we
could ensure the success of such a crucial project
with a short timeline.
There’s good news: Getting buy-in from business
teams, and especially business leadership, is
becoming commonplace. Executive management
teams replaced operations as the leading department
driving adoption of business intelligence programs in
2018 and remain a top driver of BI in 2019.
5. SUPPORT YOUR TEAM
Build projects around motivated
individuals. Give them the
environment and support they
need and trust them to get the job done.
Data warehouse adaptation:
Invest in training your team to work with
the speed and independence an Agile
process requires.
While this principle holds true for any business,
it’s especially true in an Agile operation, given
the demands for quick delivery and tight-knit
teamwork. To move to an Agile process successfully,
team members must be dedicated to the project,
understand the importance of the Agile methodology,
CHAMPION GUIDES
and be adept in their use of technology and tools.
This can entail making some hard decisions, as some
team members might not be a good fit for the new
paradigm. For example, a McKinsey study identified
that to succeed in Agile, team members needed to be
adept at handling ambiguity—a trait that might not be
needed in traditional linear approaches. Daily team huddle
You can’t just demand a team become “Agile” without best practices:
investing in training to help them become so. The
company must commit to providing training resources
and support for this transition to be successful. It will
Keep them short. Under 15 minutes
be a journey. is best. Hold them every morning,
with mandatory attendance by
everyone on the team.
6. INTERACT FACE-TO-FACE
Review assignments,
accomplishments, backlogs, and
The most efficient and effective blockers by having each team
method of conveying information
member answer three questions:
to and within a development
team is face-to-face conversation.
● What did I do yesterday?
Data warehouse adaptation:
● What will I do today?
Set up in-person daily team huddles.
● What do I need help with?
This recommendation might not be popular in the
modern culture of flexible and remote work, but
nothing beats in-person daily team huddles. When
face-to-face interaction isn’t possible, modern
conferencing software (like Zoom or WebEx) and
good project management or tracking software can
be a lifesaver. As with any data warehousing project,
documentation is critical. Tools like JIRA, VersionOne,
Rally, and LeanKit can be effective—even though they
were originally designed for software development.
55

7. MEASURE PROGRESS
Working software is the primary
measure of progress.
Data warehouse adaptation:
Recognize incremental additions as
deliverables, rather than waiting for the
finished product.
Software-focused definitions may not fit business
intelligence needs. For example, what constitutes
“working software” in business intelligence and data
warehousing? Instead of only referring to a finished
product, working software could refer to small changes
like adding a column to a table or a new calculation to
a report. A complex transformation in the form of ETL
code should also count as working software.
This shift in mindset can have a positive impact on
the business. As discussed previously, reporting
incremental deliverables to stakeholders keeps the
overall product on track, creates a window of time to
incorporate changing requirements into the plan, and
boosts employee morale by allowing them to celebrate
success regularly.
8. PROMOTE
SUSTAINABLE DEVELOPMENT
Agile processes promote
sustainable development. The
sponsors, developers, and users should be
able to maintain a constant pace indefinitely.
Data warehouse adaptation:
Sustainable ongoing development is even
more necessary in a data warehouse. It is a
long term journey which is NEVER truly done.
CHAMPION GUIDES
Data warehousing is a program, not a project. Once
a company embarks on the data warehousing path, it
can never truly be “done.” As a company grows and
evolves, its data, data sources, and data requirements
and business goals will evolve as well.
Since data warehousing programs last a long time,
practice good planning and scope control to avoid
unreasonable deadlines and employee burnout.
It’s called a “planned sprint” for a reason, so teams
shouldn’t agree to deliverables that require long hours
on an ongoing basis.
To promote sustainable development, define the
smallest valuable unit of work possible to meet the
business requirement for any given sprint or iteration.
In Agile development, this is the minimal viable
product (MVP). It may not be everything the business
team has requested, but is it better to have most of
the request delivered quickly versus all of it in a longer
period of time. After all, the goal is to deliver value
incrementally, rather than in a “big bang” waterfall
approach.
To do this, use a standard methodology or discipline,
and create a cadence of work. Track estimates
and compare to actuals. Think of it like a factory
production line where people work at a certain pace
and create product outputs predictably. If you set up
your data program right, you can measure deliverables
like a factory does. For example, you can measure how
long it takes to add a new source system, or a new
dimension or report. (Don’t be discouraged: It takes
many repetitions to get to this level of predictability!)
.
6
 CHAMPION GUIDES
Likewise, if the report writers attend the ETL code
9. CONDUCT FREQUENT walkthrough, they learn about the logic and business
In work as in life, how do you get something done
faster? It’s simple: Stop doing things you don’t need
DESIGN REVIEWS rules needed to transform the source data to useful to do. Implementing this in practice, though, can
information for business decisions. This results in be difficult.
faster handoffs and higher-quality results across the
Eliminating manual coding is the number one place
Continuous attention to design and development process.
technical excellence and good you can reduce work in data warehouses. This is
design enhances agility. where solutions like code generators are useful.
They will help you avoid syntax errors, make design
Data warehouse adaptation:
10. KEEP IT SIMPLE reviews more seamless, and help facilitate overall
Ensure the whole team is involved in flexibility, because these tools are typically platform
every design review. agnostic. And, if you choose well, they will generate
Simplicity, or the art of all the system documentation you need.
maximizing the amount of work
not done, is essential.
Traditional Agile software development teams talk
about peer review of code. In data warehousing, you Data warehouse adaptation:
need to review the design of the data warehouse— Eliminate manual coding to reduce errors
the conceptual or logical models and the schema and make revisions easier.
designs of the database. If the initial design is wrong,
then the source target mapping is going to be wrong,
which means the ETL is going to be wrong…and so
on until the end reports don’t deliver the right data.
Frequent design reviews are a must, and they should
involve the whole team. The data modeler, the ETL
developer, the business owner, and the project
manager can all learn new vocabulary and skills
from their peers through these reviews, and they
can understand how various rules are applied. If
the ETL developer is part of the data model design
review, then the developer learns about the target
model before writing code to populate that model.

7

11. ENABLE
SELF-ORGANIZING TEAMS
The best architectures,
requirements, and designs emerge
from self-organizing teams.
Data warehouse adaptation:
Challenge your team members to take on
roles based on skill sets and interest, rather
than just what’s stated in their job titles.
A uniquely Agile approach to teamwork asks team
members to figure out among themselves who is
best at what task. This requires leaders to avoid
micromanaging. Over time, team members will
naturally gravitate to roles based on skills, interest,
and personality. This approach has the added benefit
of eliminating delays by having shared responsibilities
and no single point of failure. And it supports principle
number 5, keeping your team motivated by letting
them do work they enjoy and are good at.
Groups succeed or fail as a team. If you can get a
group to agree to that, it will free them up to be less
afraid of taking risks and trying something different.
12. REFLECT ON
EFFECTIVENESS
At regular intervals, the team
reflects on how to become more
effective, then tunes and adjusts
its behavior accordingly.
Data warehouse adaptation:
Retrospectives are an indispensable
part of high-performing data and
business intelligence teams.
Team reflections in Agile are called retrospectives.
Retrospectives are a must for any business intelligence
team, regardless of the methodology they use. They
occur at the end of each sprint, iteration, or delivery
release. Done properly, they lead to an ever-improving
and efficient development process, and thus enable
the team to deliver even more high quality data and
analytics in the same timeframe.
CHAMPION GUIDES
There are three questions to review in every
retrospective:
1. What went right?
2. What went wrong?
3. What will we do differently in the next sprint?
Empower your team to lead this process. Keep in
mind, you won’t be able to fix everything all at once.
Prioritize changes to your approach just like you do
your user stories and attack the most important ones,
with the most value, in the next sprint. Have patience
with the retrospective process, as it typically takes
time for teams to master the idea and be truly open to
changing their ways.
.
8
 CHAMPION GUIDES
CONCLUSION
Agile concepts can be applied successfully to data
warehousing and business intelligence programs, but
doing so requires creativity and patience. Although
Agile development champions speed, transitioning
to an Agile process won’t happen overnight. With
patience, persistence, and the right adaptations,
Agile methodologies can hold the secret to delivering
greater business value to customers more quickly.

9

THE 12 PRINCIPLES OF AGILE
ADAPTED FOR DATA WAREHOUSING
1. Our highest priority is to satisfy the customer through
early and continuous delivery of valuable software.
Data warehouse adaptation: Identify a proxy for the
end customer if needed. That person or proxy must be
able to define what constitutes success.
2. Welcome changing requirements, even late in
development. Agile processes harness change for the
customer’s competitive advantage.
Data warehouse adaptation: Establish clear user stories
for every requirement. Leverage code generators, Agile
data engineering techniques, and a data platform that
can grow and change with you to stay nimble.
3. Deliver working software frequently, from a couple of
weeks to a couple of months, with a preference to the
shorter timescale.
Data warehouse adaptation: Focus on shortening
your delivery iterations.
4. Business people and developers must work together
daily throughout the project.
Data warehouse adaptation: Ask business owners to
join your daily meetings to ensure continued buy-in.
5. Build projects around motivated individuals. Give
them the environment and support they need and
trust them to get the job done.
Data warehouse adaptation: Invest in training your
team to work with the speed and independence an
Agile process requires.
6. The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.
Data warehouse adaptation: Set up in-person
daily team huddles.
7. Working software is the primary
measure of progress.
Data warehouse adaptation: Recognize incremental
additions as deliverables, rather than waiting for
the finished product.
8. Agile processes promote sustainable development.
The sponsors, developers, and users should be able to
maintain a constant pace indefinitely.
Data warehouse adaptation: Sustainable ongoing
development is more necessary in a data warehouse.
It is a long term journey which is NEVER truly done.
CHAMPION GUIDES
9. Continuous attention to technical excellence and
good design enhances agility..
Data warehouse adaptation: Ensure the whole team
is involved in every design review.
10. Simplicity, or the art of maximizing the amount of
work not done, is essential..
Data warehouse adaptation: Eliminate manual
coding to reduce errors and make revisions easier.
11. The best architectures, requirements, and designs
emerge from self-organizing teams.
Data warehouse adaptation: Challenge your team
members to take on roles based on skill sets and interest,
rather than just what’s stated in their job titles.
12. At regular intervals, the team reflects on how to
become more effective, then tunes and adjusts its
behavior accordingly
Data warehouse adaptation: Retrospectives are an
indispensable part of high-performing data and
business intelligence teams.
10
 ABOUT THE AUTHOR
Kent Graziano is the Chief Technical Evangelist for Snowflake Inc. He is an award-winning author, and recognized
expert in the areas of data modeling, data architecture, and Agile data warehousing. He is an Oracle ACE Director
- Alumni, member of the OakTable Network, a certified Data Vault Master and Data Vault 2.0 Practitioner
(CDVP2), expert data modeler, and solution architect with more than 30 years of experience.

ABOUT SNOWFLAKE
Snowflake is the only data warehouse built for the cloud, enabling the data-driven enterprise with instant
elasticity, secure data sharing, and per-second pricing across multiple clouds. Snowflake combines the
power of data warehousing, the flexibility of big data platforms, and the elasticity of the cloud at a fraction
of the cost of traditional solutions. Snowflake: Your data, no limits. Find out more at snowflake.com.

© 2019 Snowflake, Inc. All rights reserved. snowflake.com #YourDataNoLimits