Learn.

Connect.
Explore.
 BRK 3347

Inside Microsoft Azure

Datacenter Hardware and
Software Architecture
Mark Russinovich
CTO, Azure
@markrussinovich
Agenda Inside Azure Datacenters
Inside Azure Physical Networking
Inside Azure Logical Networking
Inside Azure Servers
Inside Azure Compute
Inside Azure Storage
Inside Azure Data Platforms
Inside Azure Datacenters
Geos and regions
The world is divided
into geographies
Defined by geo-political
boundaries or country borders
Defines the data residency
boundary for customer data
A region is defined
by a bandwidth and Region 1
latency envelope
2ms latency diameter (round trip)
Customers see regions, not DCs
100’s of miles
Different fault and flood zones,
electrical grid, hurricane zone
Typically hundreds of miles apart
Region 2
Availability Zones | intra-region resilience

AZs provide for HA in the face of Region

localized DC and software failures
Customer promise: a failure in one AZ should not
cause other AZs within the same region to fail AZ1 AZ2 AZ3 AZ4
Different water, power lines, network, generators
Customers can do application-level synchronous
replication between AZs
Three is enough for quorum

Regions offer multiple Availability Zones

(AZs)
AZ1 AZ2 AZ3 AZ1 AZ2 AZ3
At least three AZs
An AZ consists of one or more datacenters
Subscription 1 Subscription 2
Quincy, WA
Cheyenne, WY
Cheyenne, WY
Dublin, Ireland
Dublin, Ireland
Project Natick
 Environmental challenges to reliability
Can a “Lights Out” environment improve reliability?

Dust and gaseous Temperature and

People Oxidation and moisture contamination thermal cycling
Reliability Prediction of Electronic
Equipment MIL-HDBK-217 (1965)
Inside Azure
Physical Networking
 Azure networking

Azure Region ‘A’ Regional

Network Cable

Regional CDN Consumers

Network

Edge Carrier
Microsoft
WAN Enterprise, SMB,
mobile
Azure Region ‘B’
ExpressRoute
Regional
Network Internet
Exchanges
Regional Enterprise
Network DC/Corpnet

DC Hardware Services Intra-Region WAN Backbone Edge and ExpressRoute CDN Last Mile
• SmartNIC/FPGA • Virtual Networks • DC Networks • Software WAN • Internet Peering  • Acceleration for • E2E monitoring
• SONiC • Load Balancing • Regional Networks • Subsea Cables • ExpressRoute  applications and (Network Watcher,
• VPN Services • Optical Modules  • Terrestrial Fiber content Network Performance
• Firewall • National Clouds Monitoring)
• DDoS Protection
• DNS & Traffic
Azure inter-DC dark fiber backbone
DCs and network sites not exhaustive Owned capacity Data center
Leased capacity
Svalbard
Future capacity Edge site
Greenland

United States
Sweden
Norway
Canada United
Kingdom
Poland
Ukraine Kazakistan
France Russia

Turkey
United States

Iran China
Algeria Pacific Ocean
Atlanta Ocean Saudi
Libya Egypt
Mexico Arabia India Myanmar
(Burma)
Niger
Mali Sudan
Chad
Pacific Ocean Nigeria
Venezuela Ethiopia

Colombia Dr Congo
Indonesia
Peru
Angola Indian Ocean
Brazil Zambia
Bolivia
Nambia

Australia
South
Africa

Argentina
RNG regional architecture

Region
Contiguous geographical area up to roughly
100km in diameter (2.0ms RTT)
Microsoft
Regional network gateway backbone
Massively parallel, hyper scale
DC interconnect
Space and power protected
RNG RNG
Data centers
Small, Medium, or Large (T-shirt sizes)
Only contains server racks, DC network DC DC DC DC
RNGs are sized to support growing the region
by adding data centers DC DC

DC Region
Azure Network Emulator
Azure
What it is
Containerized router VMs linked via VXLAN tunnels to create
a faithful replica of production network
“Bug compatible” emulation of production network gives
S1 S2
network engineers realistic test environment
Virtual links
Status VM A Orchestrator
Used daily to de-risk major network operations
SONiC
L1 L2
T1 T2 T3 T4
Over 12 million core-hours spent on
emulation in last six months
VM B VM C
Numerous bugs caught before hitting
production network
Inside Azure
Logical Networking
Hyperscale SDN | Building the right abstractions

Proprietary appliance SDN Key to flexibility and scale is Host SDN

Management Azure Resource Manager Management plane Create a tenant

Management plane Plumb tenant ACLs

Control plane
Control to switches

Data plane Apply ACLs to flows

Data Controller

Control plane

Switch (Host)

Data plane
Azure architecture

Azure Portal CLI 3rd party

Telemetry & Insights

Azure Resource Manager

Authentication

RBAC
Service Fabric AKS Web Apps … PaaS offering

Compute Networking Storage … Resource

Provider

Azure Fabric Controller

Hardware Manager

Azure Infrastructure
Azure SDN architecture
Azure SDN Azure Resource Manager
The basis of all NW virtualization

Application

Application

Application
Containers

Containers

Containers
in our datacenters
Network RP Compute RP
VNet
Regional Network Compute
The logical network for all workloads Manager Controller CA CA CA
regardless of chosen service model or
application container Network State
Manager

Decoupled Directory Software load Node Agent

Service balancer
SDN allows compute to evolve and
converge to a single allocator
Network
Manager Agent Virtual
Filtering
Load balancer Platform
Hardware Manager agent

Azure Infrastructure One-fleet Host

Azure Front Door Service
User
HTTP(S)
Application acceleration at Microsoft’s edge sites
DDoS and application layer security
Integrated with app services and Azure websites Azure Front Door
/*
Path based traffic load balancing
Service
Global HTTP load balancing with instant failover SSL termination at Static content caching
64+ global edge POPs
Global DDoS protection and app layer security
Massive SSL offload Active global Global
traffic routing Private
WAN
Supports Microsoft’s biggest internal enterprise
and consumer apps for past 5 years probe probe

Connection
App Service Plan pooling App Service Plan

Azure Region 1 Azure Region 2

Data Replication
Azure Virtual WAN | Problem statement

End users access cloud resources directly

Corporate WAN is overloaded
On-prem security appliances strained IaaS/PaaS Business SaaS Consumer SaaS

Users circumvent IT
controls and access 2
apps directly

Security Stamps Dramatic traffic increase

strains network capacity
1 and security stamps

VPN WAN

Remote
Remote Workers Headquarters Operations
Azure Virtual WAN | General Availability

Secure connectivity for roaming

users, branch offices and head
Route Service
offices to IaaS, PaaS and SaaS
services IaaS/PaaS Business SaaS Consumer SaaS

One touch connectivity from

branch offices via integration VPN Firewall

with SDWAN vendors Azure Virtual WAN

ExpressRoute
Hub Hub Hub

Remote Workers Headquarters Remote

Operations
Azure Firewall | General Availability
Stateful Firewall as a Service L3-L7 Connectivity Policies

Deployed in a click; fully integrated with the

Azure Platform Spoke 1 NAT, Network and
Application traffic filtering
Built-in High Availability and Auto Scale rules allows
Inbound/Outbound access
Network and Application level traffic filtering
rules; NAT support
Centralized policy enforcement across VNets Central
and Subscriptions VNet

Complete VNET protection

Spoke 2
Filter Outbound, Inbound, Spoke-Spoke & Internet
Hybrid Connections traffic

Centralized Logging Traffic is denied by default

Azure to On-Prem
Archive logs to a storage account, stream traffic filtering
events to your Event Hub, or send them to
Log Analytics Spoke VNets On-Premises
Project Swift | Fast network provisioning for orchestrators

Orchestrators inject workload

in customer’s vNET
On existing VM Network Control Plane
Low provision duration

Run workload from multiple Cluster Orchestrator OrchAgent Container 1

customers on same VM

…
DNC CNI/CNS
Multiple orchestrators Master Node VM Worker Node VM
Container n

Antares, ACI/K8s, SF Mesh/SF

Network Host Agent Network Host agent

Multiple container runtime Azure Server Azure Server

Docker/Drawbridge/Argon/Xenon/Cle
arContainer
Demo
Swift
Inside Azure Servers
Azure servers | General purpose 3x
Beast

RAM
Cores
1

1 2
1 1 11
1 12
2
3
4 234 2
3
4 32 3 2

Gen 2 Gen 3 Gen 4 Godzilla Gen 5 Gen 6 Beast Beast v2

2 x Skylake 24 Core
Processor 2 x 6 Core 2.1 GHz Processor 2 x 8 Core 2.1 GHz Processor 2 x 12 Core 2.4 GHz Processor 2 x 16 Core 2.0 GHz Processor 2 x 20 Core 2.3 GHz Processor Processor 4 x 18 Core 2.5 GHz Processor 8 x 28 Core 2.5 GHz
2.7GHz
Memory 768GiB DDR4 Memory 4096 GiB Memory 12 TiB
Memory 32 GiB Memory 128 GiB Memory 192 GiB Memory 512 GiB Memory 256 GiB
Hard Drive None Hard Drive None Hard Drive None
Hard Drive 4 x 2 TB Hard Drive None Hard Drive None
Hard Drive 6 x 500 GB Hard Drive 1 x 4 TB 4 x 960 GB M.2 SSDs
SSD 4 x 2 TB NVMe, 1 x 4 x 2 TB NVMe, 1 x
6 x 960 GB PCIe Flash and 1 x 960 GB SATA SSD SSD
SSD None SSD 5 x 480 GB SSD 4 x 480 GB SSD 9 x 800 GB SSD 960 GB SATA 960 GB SATA
and 1 x 960 GB SATA NIC 40 Gb/s
NIC 1 Gb/s NIC 10 Gb/s NIC 40 Gb/s NIC 40 Gb/s NIC 40 Gb/s + FPGA FPGA Yes NIC 40 Gb/s NIC 50 Gb/s
Azure servers | General purpose 3x
Beast

RAM
Cores

4 4 4 3 3 2

Gen 2 Gen 3 Gen 4 Godzilla Gen 5 Gen 6 Beast Beast v2

2 x Skylake 24 Core
Processor 2 x 6 Core 2.1 GHz Processor 2 x 8 Core 2.1 GHz Processor 2 x 12 Core 2.4 GHz Processor 2 x 16 Core 2.0 GHz Processor 2 x 20 Core 2.3 GHz Processor Processor 4 x 18 Core 2.5 GHz Processor 8 x 28 Core 2.5 GHz
2.7GHz
Memory 768GiB DDR4 Memory 4096 GiB Memory 12 TiB
Memory 32 GiB Memory 128 GiB Memory 192 GiB Memory 512 GiB Memory 256 GiB
Hard Drive None Hard Drive None Hard Drive None
Hard Drive 4 x 2 TB Hard Drive None Hard Drive None
Hard Drive 6 x 500 GB Hard Drive 1 x 4 TB 4 x 960 GB M.2 SSDs
SSD 4 x 2 TB NVMe, 1 x 4 x 2 TB NVMe, 1 x
6 x 960 GB PCIe Flash and 1 x 960 GB SATA SSD SSD
SSD None SSD 5 x 480 GB SSD 4 x 480 GB SSD 9 x 800 GB SSD 960 GB SATA 960 GB SATA
and 1 x 960 GB SATA NIC 40 Gb/s
NIC 1 Gb/s NIC 10 Gb/s NIC 40 Gb/s NIC 40 Gb/s NIC 40 Gb/s + FPGA FPGA Yes NIC 40 Gb/s NIC 50 Gb/s
Azure servers | General purpose 3x
Beast

RAM
Cores

2 0.00000000533
1
Beasts
4 4 4 3 3 2 1

Gen 2 Gen 3 Gen 4 Godzilla Gen 5 Gen 6 Beast Beast v2 Azure Sphere
2 x Skylake 24 Core
Processor 2 x 6 Core 2.1 GHz Processor 2 x 8 Core 2.1 GHz Processor 2 x 12 Core 2.4 GHz Processor 2 x 16 Core 2.0 GHz Processor 2 x 20 Core 2.3 GHz Processor Processor 4 x 18 Core 2.5 GHz Processor 8 x 28 Core 2.5 GHz Processor 1 x A7 Core @ 500 MHz
2.7GHz
Memory 768GiB DDR4 Memory 4096 GiB Memory 12 TiB
Memory 32 GiB Memory 128 GiB Memory 192 GiB Memory 512 GiB Memory 256 GiB Memory 4 MiB
Hard Drive None Hard Drive None Hard Drive None
Hard Drive 4 x 2 TB Hard Drive None Hard Drive None
Hard Drive 6 x 500 GB Hard Drive 1 x 4 TB 4 x 960 GB M.2 SSDs Hard Drive BYO
SSD 4 x 2 TB NVMe, 1 x 4 x 2 TB NVMe, 1 x
6 x 960 GB PCIe Flash and 1 x 960 GB SATA SSD SSD
SSD None SSD 5 x 480 GB SSD 4 x 480 GB SSD 9 x 800 GB SSD 960 GB SATA 960 GB SATA SSD BYO
and 1 x 960 GB SATA NIC 40 Gb/s
NIC 1 Gb/s NIC 10 Gb/s NIC 40 Gb/s NIC 40 Gb/s NIC 40 Gb/s + FPGA FPGA Yes NIC 40 Gb/s NIC 50 Gb/s WiFi 2.4/5.0 GHz 802.11 b/g/n
Demo
Beast v2
Azure servers | Special purpose

RAM
Cores
1
1
1
1 1 1
1

HPC GPU Gen 5 HB HC NDv2 Lv2

2 x Skylake 24 Core
Processor 2 x 12 Core 2.4 GHz Processor 2 x 8 Core 2.6 GHz Processor 2 x 32 Core 2.5 GHz Processor 2 x 24 Core 3.4 GHz Processor
2.7 GHz Processor 2 x 32 core 2.0GHz
Memory 256 GiB Memory 768 GiB
Memory 128 GiB Memory 240 GiB Memory 352 GiB Memory 1 TiB
Hard Drive 1 x 2 TB Hard Drive None
Hard Drive 5 x 1 TB Hard Drive None Hard Drive None Hard Drive None
SSD 1 x 960 GB SATA SSD 6 x 960 GiB NVMe
SSD None SSD 2 x 960 GB NVMe SSD 2 x 960 GB NVMe SSD 12 x 2 TB NVMe
NIC 40 Gb/s NIC 40 Gb/s
NIC 10 Gb/s IP, 40 Gb/s IB 50 Gb/s Ethernet, 100 50 Gb/s Ethernet, 100
GPU 2 x 2 Compute GPU NIC NIC GPU 8 GPU with NVLink NIC 40 Gb/s
Gb EDR IB Gb EDR IB
Lv2 | NVMeDirect

NVMe Driver (VM)

Secure High performance

Completion
Interrupt
vRegisters

Admin CQ
Admin SQ
Virtual

Virtual
Virtualized NVMe Direct Access to NVMe

Data Queues SQ/CO n

Data Queues SQ/CO 1

Data Queues SQ/CO 2

Admin Queue for device I/O Queues
Submission Completion
filtering unsafe admin Filter Filter
commands issued Register
Physical Admin

Physical Admin
from the VM Filter

CQ
SQ

Poll for
Completion

Registers
Physical NVMe Controller
Demo
Lv2 NVMeDirect
Inside Azure Compute
Azure architecture Workloads
Web services, IoT, ML,
Microservices,
Serverless…
Azure Portal CLI 3rd party
Telemetry & Insights

Authentication
Azure Resource Manager

RBAC Service Fabric AKS Web Apps … PaaS offering

Compute Networking Storage … Resource

Provider

Azure Fabric Controller

Hardware Manager

Azure Infrastructure
Azure compute architecture
Global ARM (Azure Resource Manager)
NRP is
pass-through

Regional CRP Request

NRP RNM RDS USLB
RNM gets
network info from NSM pushes
resources TM CA:PA mappings
to RDS
RNM CDS pulls
from RDS NSM pushes VIP
makes
ranges to USLB
network
object SLB finds its
updates VIP ranges
from USLB

Inventory sync Inventory sync

Cluster TM DCM NSM CDS SLB
Inventory sync LBProgramming

Ntwk
AllocateNtwkResources programming SLBHP is
configured
Send goal Send goal Send goal with SLB
state state state NMAgent pulls from CDS endpoint

Tenant Datacenter Load

Network
Node Manager Manager
agent
agent balancer
agent
agent
Azure Serial Console

Access VM through COM1 serial port Portal

Linux Terminal Web Console Window

GRUB and emergency mode access
SysRq and NMI calls

Windows access
infrastructure
SAC access, which allows for CMD and PowerShell prompts
NMI calls Host

Enable/disable at subscription level Host

Process
Accessibility
Integrated with screen reader functionality
COM1 COM1

VM1 VM2
Demo
Serial Console
Virtual Kubelet on AKS

Virtual Kubelet uses Azure Container Kubernetes API

Instances
kubelet kubelet kubelet kubelet Virtual
ACI manages underlying compute kubelet
infrastructure Node Node Node Node

Typical kubelets implement the pod and

Only charged by the second for each container operations for each node as usual.
running container

Supports Linux and Windows containers Virtual kubelet registers itself as a “node” and allows developers to
deploy pods and containers with their own APIs
Takes advantage of both the capability of
Kubernetes and the management value Capacity NewConditions

and cost benefit of container instances OperatingSystem Virtual GetPods

CreatePod kubelet GetPodStatus
IUpdatePOD GetPod
Service Fabric

A scalable, highly-available framework

for building cloud applications

Cluster
Rich Microservice lifecycle management
Support for stateful microservices
Built-in state management via Reliable
Collections and Actors

In production
Core Azure RPs
Cortana
Cosmos DB
Skype for Business
Service Bus
Event Hubs
Event Grid
Power BI
Azure DB SF Application
Intune
…
Service Fabric Mesh
Control Plane

Azure LB Azure Portal CLI 3rd party

Azure Resource Manager

Routing service
Serverless platform
Fully managed Service Fabric RP
Container Container
Clusters are hidden
Container

service 1 service 2
service 2
Micro-billing by seconds Inventory Manager
App1 in isolated network

Containerized
Cluster Allocator
Windows and Linux
Stateful Service Fabric apps

Public preview soon

Compute Plane
Reliable collection in
any language Service Fabric Cluster 1 Service Fabric Cluster n
(5,000 nodes, 80,000 non-CPU cores) (GPU cores)
Scale on-demand
…
Demo
Service Fabric Mesh
 IoT Edge microservice clusters

ARM
Deploy SF and K8s applications
to edge clusters App model
Edge Other
IoT hub Backend
apps services
Supports stateful services

Supports high availability

Other
IoT hub
Includes built-in block driver RP
SF RP … resource Management plane
providers

Sensor/control

Service Other
IoT Hub Fabric Mesh Services Compute plane

Edge Cloud
Demo
IoT Edge high availability
Cloud data threats

Malicious Hackers Third-party

privileged exploiting access without
admins or bugs in the customer
insiders infrastructure consent
Cloud data protection

At rest In transit In use

Examples Examples Examples

Azure Storage Service HTTPS Trusted Execution Environments (TEEs)
Encryption for Data at Rest TLS Homomorphic encryption
SQL Server Transparent
Database Encryption (TDE)
Trusted Execution Environments (TEEs)

Protected container
App App
Isolated portion of processor & memory
Code & data cannot be viewed or modified from outside
Supports attestation: proving of identity Operating System Code
Supports sealing: persisting secrets
Data
Examples Hypervisor
Intel SGX
Virtualization Based Security (VBS) aka Virtual Secure Mode Hardware
Azure Confidential Computing
Encrypted
query
Confidential Computing Vision
Trusted SQL
SQL Server
Data is fully in control of the customer Module Client
Azure has no access to customer data Encrypted
Code accessing customer data is authorized by the customer results

Confidential Computing Approach Operating System

Working with silicon partners to enable confidential computing
Tools to deploy, manage and develop TEE applications
Hypervisor
Services to support TEE attestation
Confidential PaaS and SaaS services
Hardware
Demo
Multi-Party machine learning
Inside Azure Storage 
Azure Storage architecture
Blob Queue File SMB RAGRS Blob

LB LB

Front-Ends Front-Ends

Partition Layer Partition Layer

Geo replication

DFS Layer DFS Layer

Intra-stamp replication Intra-stamp replication

Storage Stamp Storage Stamp

Azure Data Lake Storage Gen 2

Blob API Gen2 API

No-compromises Data Lake

Secure Hierarchical File System
Performant
Performance Scale and Cost
Massively-scalable Security
Enhancements effectiveness
Data Governance

Cost and scale of object storage

Performance and analytics
Blob Storage
feature set of data lake storage
Object Tiering and
AAD Integration, RBAC, HA/DR support through
Lifecycle Policy
Storage Account Security ZRS and RA-GRS
Management
High Throughput Block Blob

FE BE Disk
Block Blob write throughput of 50 Gbps
Up from 60 Mbs
Blob A
Instantaneous write throughput
No ramp up

No API changes or client-side changes needed

Ingress throughput become automatically available for
block blobs
Premium Blob Storage
GB cost

Azure object storage backed by solid-state drives Premium

High throughput and low latency object storage Hot

Block blobs and Append blobs
Cool
Consistent API
Archive
Example use cases
Low
Interactive workloads like media editing
Telemetry data
Data transformation

Average Latency
Object-level tiering to Hot is on the roadmap 
Hot
(10-20ms)
Premium
(4-8ms)
Standard SSD

Standard HDD Standard SSD Premium SSD

Scale/Perf Improvements Managed Disks Managed Disks Managed Disks
Uncached Read: ~4ms
Double digit Single digit
Disk latency Uncached Write: ~2ms
milliseconds milliseconds
 ~1ms (Q4 CY2018)

Max disk size 4TB  32TB 4TB  32TB 4TB  32TB

Max disk IOPS 500 IOPS 500 IOPS 7,500 IOPS

Max disk bandwidth 60 MBps 60 MBps 250 MBps

Project Direct Drive | Ultra SSD

Client talks directly to storage,

hence the name Direct Drive
On-premises VM Disk
performance at scale

Fully configurable Low Latency High IOPS High throughput

Capacity up to 64* TB < 1ms up to 160,000 up to 2000 MB/s
IOPS per disk per disk
Variable IOPS up to 160,000
Variable throughput up to 2000 MB/s
Demo
UltraSSD
Inside Azure Data Platforms
DB System Topology at different scopes

Planet Earth Azure regions Datacenters Stamps Fault domains

Container Resource RSM

Manager
… Resource Governor
Language
Index Manager

Transport Runtime(s) Storage Engine

… Admission control
Hosts
Log Manager
Query
Various agents Database engine Processor IO Manager

Cluster Machine Replica Database engine

Azure SQL DB Flex Scale

Read and write Read only

Relational DB for cloud

based on SQL Server RBPEX
Data Cache
RBPEX
Data Cache
RBPEX
Data Cache
RBPEX
Data Cache
Secondary Secondary Secondary
Primary Compute Compute Compute Compute

Separation of compute,
Sqlservr.exe Sqlservr.exe Sqlservr.exe Sqlservr.exe

Log
storage, and log shipping Compute
Landing Zone
(Azure Premium Storage) Log
Service

Elastic scale of compute Log Destaging

and storage independently

Log
Storage Long-Term Log Storage for PITR Cache

(Azure Standard Storage)

1 TB data file 1 TB data file 1 TB data file

Scale up to 100TB storage, Covering

RBPEX
Covering
RBPEX
Covering
RBPEX
100s of reading nodes Data Cache Data Cache Data Cache

Rapid database restore Local SSD Cache

Data Data Data
pages pages pages Resilient Buffer
RBPEX
Pool Extension

File snapshots File snapshots File snapshots Log Pathway

Data Pathway
Azure Storage
Demo
SQL DB Flex Scale
Azure Cosmos DB | Multi-master at global scale

K V
K V
K V
Writes Reads
High write & read availability  Tables Collections Graphs
99.999 SLA worldwide followers
leader
forwarder
Low write and read latency
Container Replica set
<10 ms at the 99th percentile To remote
Container
Containers
Resource resource
partition(s)
partition

Elastically scale your writes & reads, worldwide … …

… Region 1
Five well-defined Consistency Models
Resource partitions

Global distribution
Designed to support hundreds of … …

regions and billions of edges … Region 2

Exposes familiar OSS APIs … … …

(Cassandra, Mongo, Gremlin), SQL and
Table Storage … …
… Region N

Partition-set

Local distribution
Demo
CosmosDB MultiMaster
 Azure
Hardware and software innovation at scale
 Please evaluate this session
Your feedback is important to us!

The s
lid e will Th
be re is slide is
place r
d ons equired.
ite th D
rough o NOT d
Silver elet
Fox P e or alter
roduc th
tions e slide.
with
an up
dated
QR code.

From your PC or Tablet visit MyIgnite at

http://myignite.microsoft.com

From your phone download and use the Ignite Mobile App
by scanning the QR code above or visiting
https://aka.ms/ignite.mobileapp
© Copyright Microsoft Corporation. All rights reserved.