Power BI Push

Datasets

Gary Lock
June 9, 2018
Intro: Gary Lock
• Solution Architect at BlueGranite

• Microsoft Certified Big Data Professional

• Full-stack BI developer including C# and ASP.NET

• Extensive ERP and CRM experience

• Finance background and MBA with concentration

in MIS and Management

Email: glock@blue-granite.com
LinkedIn: www.linkedin.com/in/gary-lock
Twitter: www.twitter.com/garylock
Why?
• Power BI is a powerful set of tools/services with many features
• Power BI Desktop
• Power BI Service
• Enterprise/Personal Gateway
• Power BI Embedded
• Streaming Datasets
• Power BI Mobile
• Streaming Datasets are not only for IoT
• Push datasets are very easy to use and can be very useful
Agenda
• What is Streaming Data?
• Real-Time Streaming in Power BI
• Demo 1 – Query Excel file
• Demo 2 – Query SQL DB
• Demo 3 – Query SQL DB (large)
• Demo 4 – Cognitive Services – Computer Vision
• Questions
Streaming Data
Data generated continuously by many sources

• Device telemetry

• Social network feeds

• Geospatial services

• Log files

• Gaming activity
Real-Time Streaming in Power BI
Types of Real-Time Datasets

Push Streaming PubNub Streaming

Streaming Dataset Matrix
Capability Push Streaming PubNub
Max rate of ingestion 1 request/s 5 requests/s N/A – data not pushed
16 MB/request 15 KB/request into Power BI
Data throughput limit 1M rows/hour None N/A – data not pushed
into Power BI
Data stored in Power BI Yes – up to 200k rows Temporarily (1 hr) No
Build Power BI Reports Yes No No
Dashboard tiles update Yes – report visuals Yes – tiles added Yes – tiles added
in real-time pinned to dashboard directly to dashboard directly to dashboard
Dashboard tiles update No Yes Yes
with smooth animations

*Push dataset consumes storage in Power BI Service

How to Stream Data to Power BI?
Power BI REST API
Power BI REST API Limitations
• 75 max columns
• 75 max tables
• 10,000 max rows per single POST rows request
• 1,000,000 rows added per hour per dataset
• 5 max pending POST rows requests per dataset
• 120 POST rows requests per minute per dataset
• If table has 250,000 or more rows, 120 POST rows requests per hour per dataset
• 200,000 max rows stored per table in FIFO dataset
• 5,000,000 max rows stored per table in ‘none retention policy’ dataset
• 4,000 characters per value for string column in POST rows operation
To Push or Not To Push
When to use a Push dataset
• Flat data structure
• Need streaming with history
• No data gateway setup
• Data source not supported by data gateway
• Need to refresh data faster than data
gateway allows
• For a quick POC
When not to use a Push dataset
• Data requires multiple, joined tables
• Volume or velocity too large
• Do not need history
• Already have a data gateway setup with
Direct Query of data
• Microsoft Flow alternative exists
Create a Streaming (Push) Dataset
• Dataset Name
• Fields/Values From Stream
• Data types:
• DateTime
• Number
• Text
• Historic Data Analysis: Off/On
• Push URL
• Sample REST API Payload
• Raw
• cURL
• PowerShell
Demo 1: CSV Source

• Push data from CSV file into Power BI

• Use PowerShell to invoke Web Request
• Can be automated using Task Scheduler
Demo 2: SQL DB Source

• Push data from SQL DB into Power BI

• Use PowerShell to invoke SQL Command and
Web Request
• Can be automated using Task Scheduler
Demo 3: SQL DB Source (large)

• Push data from CSV file into Power BI

• Use PowerShell to invoke Web Request
• Can be automated using Task Scheduler
• Due to REST API limitations, break
request into 10k row chunks
Demo 4: Computer Vision

• Microsoft Cognitive Services

• Computer Vision API
• Tag images based on content
• Categorize images
• Identify type and quality
• Detect human faces
• Recognize domain-specific content
• Generate descriptions
• Use OCR to identify printed text
• Flag adult content
• Crop photos to use as thumbnails
Questions?

Thanks for being here!