Introducing:

Self-service data prep

with dataflows
November 2018

Adi Regev , Principal GPM

Anton Fritz, Senior PM
Achieving the digital feedback loop

Empowered employees

Employees

Transformed
Intelligent Data products and
customer Customers + Products optimized
engagement Intelligence
operations
The modern business intelligence challenges
Fragmented, incomplete data
Pulling together data from traditional and
cloud data sources and figuring out how
to enrich it is extremely difficult.
Business data has no structural or
semantic consistency
Different applications, departments, and
analysts define data in different ways,
which makes data exploration, and reuse
of data and apps extremely challenging.
Requires a team of specialists Data preparation – the most time-
consuming task in analytics BI.
Creating E2E BI solutions requires
multiple BI tools. This requires specific
knowledge of each of the tools and
complex integration to build and
maintain an E2E BI solution. Complex system design and
architecture
Traditional BI solutions require planning
and designing in advance. Items like table
structure and refresh order are complex
to design and implement.
Complex system integration
Traditional BI solutions span multiple
applications and services. Sharing data
across systems requires each system to
understand the location, structure and
meaning of the data
Traditional BI platform

Data visualization

BI modeling

Gateways and connectors

BI platform is evolving

Data visualization

BI modeling

Data prep

Gateways and connectors

 Power BI introduces self-service data-prep
capabilities
Cloud and on-premises
Self-service low code/no code In-lake transformations
connectors

Standard schema
Dataflow
Data reuse Integral part of Power BI stack
(Common Data Model)

Incremental refresh Excel like -auto re-calc

Power BI introduces dataflows
Reports & dashboards
Visualizations

Datasets
BI models

Dataflows
Data prep

Storage
Data (Azure Data Lake)

Gateways and connectors

Self-service demo walkthrough
Getting data into centralized storage leveraging dataflows
Demo recap

Sales data
Dataflow

• Clean and transformed data is

ingested into the data lake

• Data is in Common Data Model

format

• Anyone with the right

permissions can leverage
the data
Self-service demo walkthrough
Reuse data and perform in-lake computation
Demo recap

1 2 3
Sales data Sales with service calls Power BI app
Dataflow Dataflow

• Create dataflows with dataflow
editor in Power BI
• Perform transformations and data
cleansing using Power Query
Online
• Map data to the Common Data
Model
• Reuse data from others in your Connect with Power BI Desktop to
department or other departments create models, reports, and
dashboards using dataflow data.
• Perform in-lake computations
• Keep data consistent with smart re-
calc logic

• Schedule a full or incremental

refresh
The modern BI challenge
Fragmented, incomplete data
Pulling together data from traditional and
cloud data sources, and figuring out how
to enrich it is extremely difficult.
Business data has no structural or
semantic consistency
Different applications, departments, and
analysts define data in different ways,
which makes data exploration, and reuse
of data and apps extremely challenging.
Requires a team of specialists
Creating E2E BI solutions requires
multiple BI tools. This requires specific
knowledge of each of the tools and
complex integration to build and
maintain an E2E BI solution.
Complex system integration
Traditional BI solutions span multiple
applications and services. Sharing data
across systems requires each system to
understand the location, structure and
meaning of the data.
Recap of what we saw:

Power BI users can now easily create dataflows to

Reports & dashboards prepare data in a centralized storage, using a
standardized schema, ready for easy consumption,
reuse, and generation of business insights
Datasets
• Data prep capabilities in Power BI

• The dataflows that are created can be managed in PowerBI.com

Dataflows
• Create entities that conform to the Common Data Model or entities
with custom schema

• Data in the Data Lake is seamlessly integrated with Power BI

Storage
artifacts and is available for consumption in Power BI Desktop

• Available to Pro and Premium customers

But what if?

We want to leverage the data created by the dataflow

Reports & dashboards in applications or services?

Datasets

Dataflows

Storage
Your data
Azure integration
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Datasets

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Storage
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

1. Provide Power BI
with details of the
Data Lake Storage
account
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

2. Enable workspace
Admins to assign
workspaces to the
storage account
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

3. Workspace Admins
can assign
workspaces to the
storage account,
Dataflows and
Dataflow data will be
stored there.
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage
Configure Power BI to
work with your
organization’s Azure
Data Lake Storage

Just provide Power BI

with details of the Data
Lake Storage account,
and your organization's
dataflows will be stored
there
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Datasets

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Azure Data Lake Storage Gen2

Data scientists
Business analysts
Data engineers
Low/no code
Low to high code
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Datasets

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Azure Data Lake Storage Gen2

CDM folder CDM folder CDM folder

Data scientists
Business analysts
Data engineers
Low/no code
Low to high code
Enrich data using Azure services

Enriched
Sales with service call center

• CDM folder created with

dataflow in Power BI can be
processed and enriched with
Azure services

• For example, perform

sentiment analysis on service
calls data
But what if?

We want to leverage the data created by the dataflow

Reports & dashboards in other applications or services?

Or to provide an easy way to consume analytical data

Datasets created outside of Power BI, in Power BI?

Dataflows

Storage
Use case: Dynamics 365 Finance & Operations data in
Power BI

Self service
Dynamics 365 for
customizations
Finance & Operations
in Power BI

Azure Data Azure Azure SQL Azure ML

Dataflow Dynamics 365 data Factory Databricks DW

Azure Data Lake Storage Gen2

CDM folder
But what if?

We want to leverage the data created by the dataflow

Reports & dashboards in other applications or services?

Or to provide an easy way to consume analytical data

Datasets created outside of Power BI, in Power BI?

Dataflows

Storage
Data + AI professionals can use the full power of the
Azure Data Platform
Reports & dashboards

Custom LOB
+
Datasets Developer
resources

Azure Data Azure Azure SQL Azure ML

Dataflows Factory Databricks DW

Azure Data Lake Storage Gen2

CDM folder CDM folder CDM folder

Data scientists
Business analysts
Data engineers
Low/no code
Low to high code
Programmability
1 2
Power BI APIs
• We’ve added new web APIs to
Power BI to help enable many
scenarios involving dataflows.
• You can now programmatically
import Power BI dataflows or add
a reference to external dataflows
in Azure storage.
• You can connect datasets to
dataflows, refresh dataflows,
update their refresh schedules,
and more.
Model file SDK
• A variety of libraries and sample
code will be available to enable
programming scenarios.
• You can validate, create, or read
CDM model files and data
programmatically.
• You can easily read CDM Model
files from Azure Databricks or write
Databricks tables to dataflows.
3
Documentation
• Documentation for APIs, model
file creation and working with
dataflows will be available at
launch.
• Step-by-step guides for key
dataflow scenarios will be
available as well.
 Questions?

© Copyright Microsoft Corporation. All rights reserved.