Transform your data estate

with Big Data, Advanced Analytics

anid@microsoft.com
The world is changing
 Today, 80% of
AI investment
Data will grow to organizations
increased by
44 ZB in 2020 adopt cloud -first
300% in 2017
strategies
 Today, 80% of
AI investment
Data will grow to organizations
D ATA
44 ZB in 2020 CLOUD
adopt cloud -first AI
increased by
300% in 2017
strategies
 D ATA AI
Organizations that harness data,
cloud, and AI outperform
CLOUD
 Companies surveyed include well-known
enterprises across key industries

Financial services Retail

Manufacturing Consumer goods

Source: Keystone Strategy interviews Oct 2015 - Mar 2016

Organizations that harness data,
cloud, and AI outperform

Nearly double $100M in additional

operating margin operating income
They rely on a modern data estate
 Rising citizen demands are driving
transformation in government services

25% 15% 10%

of government of government transactions
organizations will (such as tax collection,
roll out cognitive welfare disbursement and
immigration control) will have
solutions by 2020
embedded analytics by 2019
of government digital
services will use bots to
personalize workflows and/or
channels through which
xx
citizens and businesses
access services by 2018
Modern government challenges

Changes in services, Large, expanding

policy, and citizen volume of data
requests require more increasing complexity
responsive agencies and storage costs

Data security,
Unstructured data privacy and regulatory
limits ability to analyze requirements are
and take action of paramount
importance
Unique government opportunities

IT infrastructure Social network Logistics Web app

optimization Public Safety analysis optimization optimization

Health Fraud Analytics Data Weather Healthcare

analysis detection Consolidation forecasting outcomes

Scientific Equipment Smart Sensor

research monitoring monitoring
constantly expanding
THE MODERN
DATA E S TAT E

HYBRID

On-premises Cloud
Private cloud

Operational databases Operational databases

Data warehouses Data warehouses
Data lakes Data lakes

Reason over any data, anywhere Flexibility of choice Security and Performance
The Azure BIG Data Landscape
AZURE
AZURE AZURE IMPORT AZURE SQL DB AZURE COSMOS DB AZURE SQL DATA WAREHOUSE POWER BI
ANALYSIS SERVICES
DATA FACTORY EXPORT SERVICE

AZURE CLI AZURE SDK

AZURE DATA LAKE AZURE AZURE AZURE ML ML SERVER AZURE
AZURE STORAGE AZURE DATA LAKE ANALYTICS HDINSIGHT DATABRICKS
DATABRICKS
BLOBS STORE

AZURE IOT HUB AZURE EVENT HUBS

AZURE AZURE AZURE

AZURE SEARCH BOT SERVICE COGNITIVE SERVICES
AZURE STREAM ANALYTICS HDINSIGHT DATABRICKS
KAFKA ON DATA CATALOG
AZURE HDINSIGHT

AZURE EXPRESSROUTE AZURE AZURE NETWORK AZURE KEY OPERATIONS AZURE FUNCTIONS
VISUAL STUDIO
ACTIVE DIRECTORY SECURITY GROUPS MANAGEMENT SERVICE MANAGEMENT SUITE
SQL Server 2019
Azure Data Lake
Azure Data Bricks
 Industry-leading performance and security, with intelligence over all your data

Intelligence over Choice of platform Industry-leading Most secure Insights in minutes

any data and language performance over the last 8 years5 and rich reports

200
180
160

Vulnerabilities (2010-2017)
140
120
100
80
60
#1 OLTP performance1 40
20
#1 DW performance on 0
AI and Machine Learning T-SQL PHP Python The best of Power BI and
1TB2, 10TB3, and 30TB4
over all data with the power Java Node.js Ruby SQL Server Reporting Services
of SQL and Apache Spark C/C++ C#/VB.NET Intelligent Query Processing with Power BI Report Server

In-memory across all workloads Most consistent data platform

Private cloud 1/10th the cost of Oracle Public cloud

All TPC Claims as of 1/19/2018.

1 http://www.tpc.org/4081; 2 http://www.tpc.org/3331; 3 http://www.tpc.org/3326; 4 http://www.tpc.org/3321; 5 National Institute of Standards and Technology Comprehensive Vulnerability Database
 Data virtualization Managed SQL Server, Spark, Complete AI platform
and data lake

Admin portal and management services

Analytics Apps
T-SQL Integrated AD-based security
REST API containers
for models

SQL Server External Tables SQL

Server Spark
SQL Server Spark &
ML Services Spark ML
Compute pools and data pools

Scalable, shared storage (HDFS)

Open NoSQL Relational HDFS External HDFS
database databases data sources
connectivity

Combine data from many sources without Store high volume data in a data lake and access Easily feed integrated data from many sources to
moving or replicating it it easily using either SQL or Spark your model training
Scale out compute and caching to boost Management services, admin portal, and Ingest and prep data and then train, store, and
performance integrated security make it all easy to manage operationalize your models all in one system
 Custom
apps BI Analytics

SQL Server
SQL
master instance

Compute pool Compute pool Compute pool Directly

read from
…
SQL Compute SQL Compute SQL Compute SQL Compute SQL Compute
Node Node Node Node Node HDFS

Data mart Storage pool

SQL Data SQL Data

Node Node Spark
SQL
Spark
SQL
… Spark
SQL
Server Server Server

HDFS Data Node HDFS Data Node HDFS Data Node

Storage Storage
Kubernetes pod
IoT data

Node Node Node Node Node Node Node

Persistent storage
 Performance
Intelligent Query Processing
Accelerating I/O performance with Persistent Memory
Gain performance insights anytime and anywhere with Lightweight Query Profiling

Security
Always Encrypted with secure enclaves
Data Classification and auditing built-in
Manage certificates easier with SQL Configuration Manager

Availability
Always On availability group enhancements
Resumable online index creation
Online Clustered Columnstore index creation and rebuild
Availability groups on Kubernetes
SQL Server 2019
Azure Data Lake
Azure Data Bricks
 VALUE
How can we
make it happen?
Prescriptive
What will Analytics
happen?
Theory
Predictive
Theory Analytics
Why did Hypothesis
Hypothesis it happen?
Diagnostic Pattern
Observation What
Analytics
happened?
Observation
Descriptive
Confirmation
Analytics

DIFFICULTY
Understand Gather Implement Data Warehouse
Corporate Requirements
Strategy Reporting & BI and analytic
Reporting &
Analytics Design Analytics
Business Development
Requirements
Data warehouse
Dimension Modelling Physical Design

ETL
ETL
ETL Design
Technical Development
Requirements
Data sources
Setup Infrastructure Install and Tune
Data Lake Uses A Bottom-Up Approach

Ingest all data Store all data Do analysis

regardless of requirements in native format without Using analytic engines
schema definition like Hadoop

Batch queries

DEVICES
Interactive queries
Real-time analytics
r
LOGS, FILES AND MEDIA Machine Learning
(UNSTRUCTURED)

Data warehouse

BUSINESS / CUSTOM
APPS
(STRUCTURED)
 WASB WASB ADLS Azure Data Lake Storage Gen2

Blob Storage + Blob Storage + Azure Data Lake

Scalable, secure storage that
(WASB) (WASB) Store (ADLS) speeds time to insight

Scale and Scale and Speed to Scale and Speed to

Availability Availability Insight Availability Insight
Cost Cost Rich Cost Rich
Effectiveness Effectiveness Security Effectiveness Security

Azure Data Lake Storage Gen2: Single Data Lake Store that combines the performance and
innovation of ADLS with the scale and rich feature set of Blob Storage
SQL Server 2019
Azure Data Lake
Azure Data Bricks
What is Azure Databricks?
A fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure

Best of Databricks Best of Microsoft

Designed in collaboration with the founders of Apache Spark

One-click set up; streamlined workflows

Interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.

Native integration with Azure ser vices (Power BI, SQL DW, Cosmos DB, Blob Storage)

Enterprise-grade Azure security (Active Director y integration, compliance, enterprise -grade SL As)
Azure Databricks
Azure Databricks
Collaborative Workspace

IoT / streaming data Machine learning models

DATA ENGINEER DATA SCIENTIST BUSINESS ANALYST

Deploy Production Jobs & Workflows

BI tools
Cloud storage

MULTI-STAGE PIPELINES JOB SCHEDULER NOTIFICATION & LOGS

Data warehouses
Optimized Databricks Runtime Engine Data exports

Hadoop storage
DATABRICKS I/O APACHE SPARK SERVERLESS Rest APIs
Data warehouses

Enhance Productivity Build on secure & trusted cloud Scale without limits
Collaborative Workspace
Azure Databricks
Collaborative Workspace

DATA ENGINEER DATA SCIENTIST BUSINESS ANALYST

Deploy Production Jobs & Workflows

MULTI-STAGE PIPELINES JOB SCHEDULER NOTIFICATION & LOGS

Optimized Databricks Runtime Engine

DATABRICKS I/O APACHE SPARK SERVERLESS Rest APIs

Deploy Production Jobs & Workflows
Azure Databricks
Collaborative Workspace

DATA ENGINEER DATA SCIENTIST BUSINESS ANALYST

Deploy Production Jobs & Workflows

MULTI-STAGE PIPELINES JOB SCHEDULER NOTIFICATION & LOGS

Optimized Databricks Runtime Engine

DATABRICKS I/O APACHE SPARK SERVERLESS Rest APIs

Optimized Databricks Runtime Engine
Azure Databricks
Collaborative Workspace

DATA ENGINEER DATA SCIENTIST BUSINESS ANALYST

Deploy Production Jobs & Workflows

MULTI-STAGE PIPELINES JOB SCHEDULER NOTIFICATION & LOGS

Optimized Databricks Runtime Engine

DATABRICKS I/O APACHE SPARK SERVERLESS Rest APIs

 D ATA M A N A G E M E N T S O L U T I O N
FOR INTELLIGENCE IN THE CLOUD

Ingest Store Prep & Train Model & Serve Intelligence

Business
apps Data Factory
(Data movement, pipelines & orchestration)

Cosmos DB

Predictive apps

Kafka Blobs HDInsight /

10
01
SQL
SQL Database
Custom Data Lake Spark
apps

SQL Data
Operational reports
Warehouse
Event Hubs Machine
IoT Hub Learning
Sensors Analysis Services
and devices
Analytical dashboards

DATA INTELLIGENCE ACTION

THE MODERN
DATA E S TAT E

HYBRID

On-premises Cloud
Private cloud

Operational databases Operational databases

Data warehouses Data warehouses
Data lakes Data lakes

Reason over any data, anywhere Flexibility of choice Security and Performance
Empower today’s innovators to unleash the power of data
and reimagine possibilities that will improve our world