The Data Warehouse

Comeback:
How the Cloud is
Accelerating Insights
David Stodder
TDWI Senior Director of Research, BI
SPEAKERS

David Stodder Noel Yuhanna Clive Bearman

Senior Research Director, Guest Speaker Sr. Director of Product Marketing,
Business Intelligence, TDWI Principal Analyst Serving Attunity, a division of Qlik
Enterprise Architecture
Professionals, Forrester
Sponsor

3
 WEBINAR

TDWI Perspective:
The Data Warehouse Comeback: How the
Cloud is Accelerating Insights
DAVID STODDER

Senior Research Director

Business Intelligence
TDWI
dstodder@tdwi.org
@dbstodder
Data Warehousing: Why the Comeback?
• More than ever, organizations need clean,
integrated, and trusted data
– For analytics, shortening path to quality data
• Data lakes: complement to DWs
– Useful as staging area or operational data store
for DWs
• Solving self-service silo headaches: On
their own, users spend too much time
manually preparing, modeling, transforming,
and improving data (60%-80% of their time)
The Cloud: Spurring a New View of DW
• Cloud platforms can bring greater flexibility,
elasticity, and speed to data warehousing
– On premises: DWs notoriously inflexible, slow to
develop and deploy
• Cloud key to adding analytical depth for more
interactive reports and dashboards
– Additional trend toward “augmented” AI-driven user
experiences requires elasticity, scale, and speed
• Reality: Hybrid, multicloud data architectures
– Data movement and migration from on-prem to cloud
Research: Cloud is Central to Data Strategies
Prime motivator: Scalability,
elasticity, and speed for
growing analytics
• Cloud-native BI platform: (38%)
– New on-premises platform: 21%
• Cloud-based data lake: 31%
• Cloud data storage: 24%
• Cloud-native DW: 21%
– New on-premises DW: 13%

Source: TDWI Research, 2018

Problem: Repeating Errors of Past DWs
• Hard to keep pace with business demands and expectations
– Slow data warehouse development and deployment cycles can
persist, even with cloud (“cloud was supposed to be fast!”)
– Hard to change after deployment
• Too much reliance on slow, inconsistent ETL development; not
enough reuse and continuous improvement
– Often 100s if not 1000s of one-off ETL routines to support
• Integrating new data is slow; heavy movement across networks
• Lack of visibility: poor monitoring of data usage, performance, and
governance
Key Ways that DW Automation Can Help
Data warehouse automation: eliminating manual work in DW
lifecycle (design, development, deployment, etc.)
• Accelerate pace of development: enable experimentation
and use of agile methods for faster business value
• Speed: less manual work and greater efficiency
– Speed of business: keep pace with trend toward business-
driven creation of virtual data warehouses and data marts
• Modeling and schema development: Automation can
address key areas that slow down cloud DW deployment
– Mapping on-premises structures to cloud-based ones
Automation for Higher Cloud DW ROI
• Take advantage of speed of deployment
– Increases reuse to shorten path to
continued development and deployment
• Enabling focus on where latency
continues to exist:
– Defining objectives and desired outcomes
– Defining bigger entities, e.g., customers
– Tools can then streamline development of
models, schema, cubes, sandboxes, etc.
DW Automation and Emerging DataOps
• Data + Ops: Enterprise collaboration framework and
methodology; improving business/tech alignment
– Iterative lifecycle: Origins in DevOps, Agile, Lean
– End-to-end: aligns data consumption with management
– Goals for control, transparency, auditability
– Improving data pipelines for agility as well as
management
• DW automation for DataOps: Supporting Credit: ZDNet

standardization, use of metadata, closer

alignment with goals for business value
Recommendations
Don’t just apply older ways to the new platform;
use migration to the cloud as an opportunity to
improve data warehouse lifecycle
 Automate processes to increase speed,
efficiency, and productivity
 Not just more ETL: match data integration
approach to the right use case
 Improve self-service; shield users from
(often manual) complexity to enable
business-driven data warehousing
Recommendations
Tap the benefits of going to the cloud by aligning data
warehousing with cloud architecture directions
 Use data warehouse automation to enable continuous
improvement and delivery of incremental benefits
• Align with use of agile and DataOps methods
 Standardize to improve governance of entire environment
(hybrid, multicloud); govern data migration and movement
 Align data warehouse development with cloud development
directions for efficiency and standards (e.g., containerization)
 TDWI

Cloud Data Warehouse Drives

Innovation and Growth

Noel Yuhanna, Principal Analyst

October 2019
© 2019 FORRESTER. REPRODUCTION PROHIBITED.
 Data is the new …

Currency Oil 12%

Companies
monetize on data

It’s driving today’s digital business strategy..

© 2019 FORRESTER. REPRODUCTION PROHIBITED. 16

Data explosion
Data means
is the new … new opportunities for every
organization …

Video social

of data is on the Cloud

sensors
public net

IOT
mobile
Data
© 2019 FORRESTER. REPRODUCTION PROHIBITED. 17
 Trends – data and analytics

› Drive towards a connected enterprise

› Distributed data is a reality – on-premises, cloud, multi-cloud

› Big focus on real-time analytics

› Focus is on self-service – from users and IT

› Security and governance becomes a priority

› Cloud data warehouse is delivering innovation and new

features.

© 2019 FORREST ER. REPRODUCT ION PROHIBIT ED. 18

 Types of workloads in the public cloud

› Analytics and operational insights dominate the public cloud

deployments
› Wide variety of use cases seen in the public and multi-cloud –
including 360-degree view of the customer, actionable insights, real-time
analytics, IoT analytics, Fraud and security risk analytics etc..
› Most of the analytics deployed in the public cloud are new insights that
haven’t been done before.
› Cloud data warehouse adoption has been growing rapidly, with most
moving their existing warehouses to the cloud, with largest being 10’s of
Petabytes.

© 2018 FORRESTER. REPRODUCTION PROHIBITED. 19

 Cloud Data Warehouse – Offers you scale and
automation..

Customer 360
Real-time Insights

IoT Analytics
• Elastic scale
• Automation
• AI/ML
• Security/Gov.
Business users • Innovation
• Lower cost
Analytics

Analytical Backup/ DR/

Data Platform High
Analytical Availability
warehouse
Platform

Data
warehouse
Public Cloud
On-premise

Admin/Data Engineers

© 2019 FORRESTER. REPRODUCTION PROHIBITED. 20

 Recommendations

› Cloud data warehouse is delivering innovation and new capabilities

› Cloud data warehouse offers new business possibilities
› Keep security and governance in mind right from the start
› Create a team of experts to ensure success – multiple personas
› Use AI/ML to automate preparation, integration, and support new use
cases.
› Leverage solutions to help accelerate cloud data warehouse
deployments, including data migrations.

© 2019 FORRESTER. REPRODUCTION PROHIBITED. 21

The Data Warehouse
Comeback:
How the Cloud is Accelerating
Insights

Clive Bearman
Director Product Marketing
New Data-driven
Opportunities

23
Traditional Data Warehouse Architecture

Data Sources Data Storage Data Consumer

Orchestration

24
Every Cloud Data Warehouse Architecture

Data Sources Data Storage Data Consumer

Orchestration

25
With Instant Benefits!

• Consumption-billing
• On-demand
• Self-service
CLOUD
• Elastic

• Billed regardless of usage

• Peak-provisioned
• IT-managed
ON PREM • 24x7 model

26
Problem Solved, Right?

27
28
 Actual Data Warehouse Architecture
Regardless of Cloud, On-premises or Hybrid
DATA LANDING STAGING DATA DATA DATA
SOURCE ZONE* ZONE WAREHOUSE MARTS CONSUMERS
S

RDBMS

DM 1 DM 2 DM 3
FILES
LOAD
API
DM 4 DM 5
MAINFRAME
</>
CDC

DM 6
APPS

SAP
Catalog, Search and Governance

29
 Actual Data Warehouse Architecture
ETL Coding and Maintenance Bottleneck
DATA LANDING STAGING DATA DATA DATA
SOURCE ZONE* ZONE WAREHOUSE MARTS CONSUMERS
S
Semi-structured 3rd NF Denormalized Star Schema

RDBMS 1000’s of tables and files 1000’s of tables 1000’s of tables 1000’s of tables

ETL DM 1 DM 2 DM 3
FILES
INGEST

ETL ETL
API
ETL DM 4 DM 5
MAINFRAME
</>
INGEST

ETL DM 6
APPS

SAP
Catalog, Search and Governance

30
 Achieving a Balance

Build Run

• Consumption-billing • Productivity
• On-demand • Flexibility
• Self-service • Maintainability
• Elasticity • Accurate Data

31
Data Warehouse Automation
Features
• Automates Data Model/Data Warehouse
design, build, maintenance and
documentation
• Automated table creation, instantiation
and mappings
• Continuous, real-time data replication

Benefits High-Scale, Multi- SQL Server Data Real-Time

sourced Data Warehouse Consolidation from
• Reduce risk, save time and money Consolidation into Automation Cut ETL SAP, Oracle to
– no scripting or coding required Azure Data Lake and scripting time 96%, Snowflake for DW
Azure SQL DW accelerated change Modernization
• DW now built in hours, changed in
process 6x
minutes
• Future-proof for new requirements and
new platforms

32
 Actual Data Warehouse Architecture
Automated Analytics-ready Data Marts
DATA LANDING STAGING DATA DATA DATA
SOURCES ZONE* ZONE WAREHOUSE MARTS CONSUMERS
Semi-structured 3rd NF Denormalized Star Schema

Error Facts/Dimension
RDBMS
Metadata

ETL DM 1 DM 2 DM 3
FILES
LOAD

ETL ETL
API
ETL DM 4 DM 5
MAINFRAME
</>
CDC

ETL DM 6
APPS

SAP
Catalog, Search and Governance

33
We budgeted for 45 days of ETL coding and
actually used 2, reduced implementation
costs 80%, and accelerated DW updates
from twice a year to at least once a month”

- Data Manager Zurich

Insurance Group

34
DataOps for Analytics

CONTINUOUS AUTOMATE CATALOG

DELIVERY REFINEMENT & GOVERN

1 Real Time Data for

Faster, Better Insights 2 Agile Data Delivery 3 Trusted, Enterprise
Ready Data

Attunity Attunity Qlik Data

Replicate® Compose® Catalyst®
35
Qlik Data Integration – Guiding Principles

Independent Universal Real-Time

Agile &
Governed Self-Service
Automated

36
Next Steps

Download the eBook:

Data Warehouse
Automation in Azure
for Dummies

37
To learn more:

qlik.com/attunity

38
Audience Q&A

tdwi.org
 Thank You!

David Stodder Noel Yuhanna Clive Bearman

Senior Research Director, Guest Speaker Sr. Director of Product Marketing,
Business Intelligence, TDWI Principal Analyst Serving Attunity, a division of Qlik
Enterprise Architecture
Professionals, Forrester
 Thank You to Our Webinar Sponsor

41
Contact Information
If you have further questions or comments:

David Stodder, TDWI

dstodder@tdwi.org

Noel Yuhanna, Forrester

nyuhanna@forrester.com

Clive Bearman, Attunity, a division of Qlik

Clive.Bearman@qlik.com