Professional Documents
Culture Documents
ACI Anywhere Extending The ACI Fabric - Max Ardica
ACI Anywhere Extending The ACI Fabric - Max Ardica
Max Ardica
Principal Engineer – Cisco INSBU
2
Agenda
Web App DB
Outside QoS QoS QoS
(Tenant Service
Filter Filter
VRF)
APIC
Application Policy
ACI Fabric Infrastructure Controller
Integrated GBP VXLAN Overlay
Ecosystem Partners
MP-BGP - EVPN
…
ACI 2.0 - Multiple Networks ACI 3.1/3.2 - Remote Leaf
(Pods) in a single Availability and vPod extends an
Zone (Fabric) Availability Zone (Fabric) to
remote locations
ACI 1.0 - ACI Multi-Pod Fabric ACI 3.0 – Multiple Availability ACI Remote Leaf Future – ACI Extensions
Leaf/Spine Single Zones (Fabrics) in a Single to Multi-Cloud
Pod Fabric IPN Region ’and’ Multi-Region
Pod ‘A’ Pod ‘n’ Policy Management
MP-BGP - EVPN
…
APIC Cluster
Application
workloads
deployed across
availability zones 19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 9
Typical Requirement
Creation of Two Independent Fabrics/AZs
ACI Multi-Site
MP-BGP - EVPN
APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP
Availability Zone
• Multiple ACI Pods connected by an IP Inter-Pod • Forwarding control plane (IS-IS, COOP)
L3 network, each Pod consists of leaf and spine fault isolation
nodes • Data Plane VXLAN encapsulation between
• Managed by a single APIC Cluster Pods
• Single Management and Policy Domain
19 - 21 March 2018 | Cisco Connect | End-to-end
•Rovinj, Croatia
BRKACI-2003 policy enforcement 12
Single Availability Zone with Tenant Isolation
Isolation for ‘Virtual Network Zone and Application’ Changes
Inter-Pod Network
ACI Multi-Pod
Fabric
APIC Cluster
• The ACI ‘Tenant’ construct provide a domain of application and associated virtual
network policy change
• Domain of operational change for an application (e.g. production vs. test)
19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 13
ACI Multi-Pod
Supported Topologies
Intra-DC Two DC sites directly connected
1G/10G/40G/100G
10G*/40G/100G 10G*/40G/100G
POD 1 POD n POD 1 10G*/40G/100G
Dark fiber/DWDM
10G*/40G/100G
POD 2
(up to 50** msec RTT)
…
MP-BGP - EVPN
Multi-Site Orchestrator
Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• ACI Multi-Site Orchestrator pushes cross-fabric • Data Plane VXLAN encapsulation across
configuration to multiple APIC clusters providing sites
scoping of all configuration changes • End-to-end policy definition and enforcement
19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 16
ACI Multi-Site
Main Use Cases
MP-BGP - EVPN
VNID à 16678781
Class-ID: 49153 EP1 EP2 Site 2
Site 1 EPG C EPG
Leaf to Leaf VTEP, Class-ID is local to the Fabric
Leaf to Leaf VTEP, Class-ID is local to the Fabric
• Maintain separate name spaces with ID translation performed on the spine nodes
VNID Class-ID Tenant Packet
•
VNID Requires specific HW on the spine to support for this functionality
Class-ID Tenant Packet VNID Class-ID Tenant Packet
§ Support all ACI leaf switches (1st Generation, -EX and -FX)
§ Only –EX spine (or newer) to Can have only a subset
of spines connecting to
connect to the inter-site network IP Network
the IP network
§ Bridge Domains and subnets not § Same IP subnet defined in separate § Interconnecting separate sites for
extended across Sites Sites fault containment and scalability
reasons
§ Layer 3 Intra-VRF or Inter-VRF § Support for IP Mobility (‘cold’ and
communication (shared services ‘live’* VM migration) and intra- § Layer 2 domains stretched across
across VRFs/Tenants) subnet communication across sites Sites, support for ‘live’* VM
migration and application clustering
§ No Layer 2 BUM flooding across
sites § Layer 2 BUM flooding across
sites
Inter-Site Network
MP-BGP - EVPN
Support planned for a future ACI release for FX line cards and 9364C platform
19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 22
ACI Multi-Site
Multi-Site Orchestrator (MSO)
• Micro-services architecture
§ Multiple MSO nodes are created and run
REST concurrently (active/active)
GUI
API
§ vSphere VM only form factor initially (physical
appliance planned for a future ACI release)
ACI Multi-Site
• OOB Mgmt connectivity to the APIC clusters
VM VM VM deployed in separate sites
§ Support for 500 msec to 1 sec RTT
Hypervisor • Main functions offered by MSO:
§ Monitoring the health-state of the different ACI Sites
§ Provisioning of day-0 configuration to establish
…..
Site 1 Site 2 Site n inter-site EVPN control plane
§ Defining and provisioning policies across sites
§ Day-2 operation functionalities
Schema
• Template = APIC policy definition
(App & Network) TemplateDEFINITION
Template
POLICY
Extending the ACI policy model outside the main datacenter to remote
sites distributed over IP Backbone (Satellites DCs, CoLo locations, etc.)
All hardware from -EX onwards is required for remote leaf nodes
and the spines to which they get associated
L3Out
vSwitch Bare
Hypervisor Metal
§ All major features supported on RL nodes
§ Scalability values are the same as for local
leaf deployments Service Graph
§ 40 RL (20 sites, 2 RL per site) (unmanaged mode) with
PBR
First-hop L3 device
receives reachability OSPF can be
information to RL redistributed into any
node other WAN control
IP WAN IPN plane
OSPF
OSPF RL node RL TEPs
configuration OSPF assigned to the
fully provisioned node
RL Node
COOP Sessions
between RLs
nodes and spines
COOP DB
EP1 RL TEP
IP WAN IPN Anycast
Anycast VTEP
EP2 RL TEP VTEP for (RL
for RLs RLs
TEP)
(RL TEP)
RL Local Table
EP1 vPC Po1
EP1 EP2
EP info sync
over vPC
IP WAN IPN control plane
EP1 EP2
IP WAN IPN
EP1 EP2
RL Site1 RL Site 2
IP WAN
Traffic between
endpoints part of
separate RL sites
always flows through
the spines
ACI Main
| CiscoDC
19 - 21 March 2018 Connect | Rovinj, Croatia 36
ACI Remote Physical Leaf and Multi-Pod
RL Sites Can Be Associated to Separate Pods
RL Site 2
RL Site 1 associated
associated to Pod2
to Pod1
RL Site1 RL Site 2
Inter-Pod Network IP WAN
IP WAN
(Multicast Enabled)
RL Site 1 RL Site 2
Inter-Pod Network IP WAN
IP WAN
(Multicast Enabled)
RL Site 1 RL Site 2
IP WAN
IP WAN Inter-Site Network
Q2
VM AVE
VM VM
VM VM VM
User
Space
AVE Policy Enforcement,
Services, Telemetry
Kernel Switching +
ACI Virtual Edge (AVE) AVS Native vSwitch
Space Policy Enforcement
VM VM VM VM VM VM VM
vSpine
vLeaf vLeaf
ACI Virtual Edge
Hypervisor
Remote
On-Premise Location
VM VM VM VM VM VM VM VM VM VM VM VM VM VM
between ‘spines’
iVXLAN Overlay
We
App DB
b
IP
Network
We
App DB
b
iVXLAN Overlay
iVXLAN Overlay
Web App DB
IP
Network
Web App DB
ACI Multi-Site
Orchestrator
Site A
Site C
“Cloud APIC (cAPIC)”
Site D
Site B
VM VM VM
VM VM VM
Network Adapter
19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 50
cAPIC and Policy Mapping
Azure Cloud Constructs
APIC
Subnet BD Subnet
Network Adapter
19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 51
ACI Anywhere
Where to Go for More Information
ü ACI Multi-Pod White Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737855.html?cachemode=refresh
ü ACI Multi-Pod Configuration Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739714.html
ü ACI Multi-Site White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739609.html
ü ACI Multi-Pod Cisco Live Barcelona 2018
https://www.ciscolive.com/global/on-demand-library/?search=weston#/session/BRKACI-2003
ü ACI Multi-Site Cisco Live Barcelona 2018
https://www.ciscolive.com/global/on-demand-library/?search=ardica#/session/BRKACI-2125
ü ACI Physical Remote Leaf White Paper
Coming soon!
19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 52