ACI Multisite Deployment - BRKACI-3502

#CLMEL
ACI Multi-Site
Architecture and
Deployment
Max Ardica – Principal Engineer (DCNBU)

BRKACI-2125
#CLMEL
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKACI-2125
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Objectives
At the end of the session, the participants should be able to:

• Articulate the different deployment options to interconnect
Cisco ACI networks (Multi-Pod and Multi-Site) and when to
choose one vs. the other
• Understand the functionalities and specific design
considerations associated to the ACI Multi-Site architecture
Initial assumption:
• The audience already has a good knowledge of ACI main
concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)
#CLMEL BRKACI-2125 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda • ACI Network and Policy Domain Evolution
• ACI Multi-Pod Quick Review
• ACI Multi-Site Deep Dive

• Overview and Use Cases
• Introducing the ACI Multi-Site Orchestrator (MSO)
• Inter-Site Connectivity Deployment Considerations
• Control and Data Plane
• Connecting to the External Layer 3 Domain
• Network Services Integration
• Virtual Machine Manager (VMM) Integration
• Multi-Pod and Multi-Site Integration
• Migration Scenarios
• Conclusions and Q&A
ACI Network and Policy
Domain Evolution
Cisco ACI: Industry Leader
5,800+ 50+% 65+

ACI Customers ACI Attach Rate Ecosystem Partners
Ecosystem Partners
ACI Anywhere
Fabric and Policy Domain Evolution
ACI Single Pod Fabric ACI Multi-Site ACI Multi-Cloud

ISN
Fabric ‘A’ Fabric ‘n’
MP-BGP - EVPN
… ACI 3.1/4.0 - Remote

ACI 2.0 - Multiple
Networks (Pods) in a Leaf and vPod extends an
single Availability Zone Availability Zone (Fabric)
(Fabric) to remote locations
ACI 1.0 - ACI 3.0 – Multiple ACI 4.1 – ACI Extensions

ACI Multi-Pod Fabric Availability Zones (Fabrics)
ACI Remote Leaf
Leaf/Spine Single to Multi-Cloud
Pod Fabric in a Single Region ’and’
IPN Multi-Region Policy
Pod ‘A’ Pod ‘n’
Management
MP-BGP - EVPN
…
APIC Cluster
Regions and Availability Zones
OpenStack and AWS Definitions
OpenStack
• Regions - Each Region has its own full OpenStack
deployment, including its own API endpoints, networks
and compute resources
• Availability Zones - Inside a Region, compute nodes can
be logically grouped into Availability Zones, when launching
new VM instance, we can specify AZ or even a specific
node in a AZ to run the VM instance
• Regions – Separate large geographical areas, each Amazon Web Services

composed of multiple, isolated locations known as
Availability Zones
• Availability Zones - Distinct locations within a region
that are engineered to be isolated from failures in other
Availability Zones and provide inexpensive, low latency
network connectivity to other Availability Zones in the
same region
Typical Requirement
Creation of Two Independent Fabrics/AZs
Fabric ‘A’ (AZ 1)
Fabric ‘B’ (AZ 2)
Application
workloads deployed
across availability
zones
Typical Requirement
Creation of Two Independent Fabrics/AZs
Multi-Pod Fabric ‘A’ (AZ 1)
‘Classic’ Active/Active
Pod ‘1.A’ Pod ‘2.A’
ACI Multi-Site
Multi-Pod Fabric ‘B’ (AZ 2)
Pod ‘1.B’Application Pod ‘2.B’

workloads deployed
across availability
zones
ACI Multi-Pod Quick
Review
ACI Multi-Pod For More Information on
ACI Multi-Pod:
Overview BRKACI-2003
VXLAN
Inter-Pod
Pod ‘A’ Network
Pod ‘n’
MP-BGP - EVPN
…
Up to 50 msec RTT
APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP
Availability Zone
• Multiple ACI Pods connected by an IP Inter-Pod L3 • Forwarding control plane (IS-IS, COOP) fault
network, each Pod consists of leaf and spine nodes isolation
• Up to 50 msec RTT supported between Pods • Data Plane VXLAN encapsulation between Pods
• Managed by a single APIC Cluster • End-to-end policy enforcement
• Single Management and Policy Domain
Single AZ with with Tenant Isolation
Isolation for ‘Virtual Network Zone and Application’ Changes
Inter-Pod
Network
ACI Multi-Pod
Fabric
APIC Cluster
Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain
• The ACI ‘Tenant’ construct provide a domain of application and associated virtual network policy
change
• Domain of operational change for an application (e.g. production vs. test)
ACI Multi-Pod
Most Common Use Cases
 Need to scale up a single ACI fabric above

Pod
200 leaf nodes supported in a single Pod
Inter-Pod
 Handling 3-tiers physical cabling layout Leaf Nodes Network
(for example traditional N7K/N5K/N2K
deployments)
Spine Nodes
 True Active/Active DC deployments

Pod 1 Pod 2
Single VMM domain across DCs (ESXi
Metro Cluster, vSphere HA/FT, DRS,…)
Deployment of clustered network services
(FWs, SLBs)
Application clustering (L2 BUM extension DB
APIC Cluster
Web/App Web/App
across Pods)
ACI Multi-Site Deep
Dive
Overview and Use Cases
ACI Multi-Site VXLAN
ACI 3.0(1)
Release
Overview Inter-Site
Network
MP-BGP - EVPN
Multi-Site
Orchestrator
Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
Region 1
• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• No latency limitation between Fabrics • Data Plane VXLAN encapsulation across
• ACI Multi-Site Orchestrator pushes cross-fabric sites
configuration to multiple APIC clusters providing • End-to-end policy definition and
scoping of all configuration changes enforcement
ACI Multi-Site
Most Common Use Cases
• Scale-up model to build a very • Data Centre Interconnect (DCI)

large intra-DC network (above • Extend connectivity and policy
400 leaf nodes) between ‘loosely coupled’ DC sites
• Disaster Recovery and IP mobility
use cases
ACI Multi-Site
Software and Hardware Requirements
• ACI Multi-Site introduced from release 3.0(1)
• Support all ACI leaf switches (1st Generation, -EX and -FX) Can have only a subset
Inter-Site of spines connecting to
• Only –EX spine (or newer) to connect to the ISN Network (ISN) the IP network
• New 9364C/9332C non modular spine

1st Gen 1st Gen -EX -EX
(64/32 40G/100G ports) also supported
• 1st generation spines (including 9336PQ)
not supported
• Can still leverage those for intra-site leaf
to leaf communication
ACI Multi-Site
Network and Identity Extended between Fabrics
Network information carried across Identity information carried across

Fabrics (Availability Zones) Fabrics (Availability Zones)
VTEP IP VNID Class-ID Tenant Packet No Multicast Requirement

in Backbone, Head-End
Replication (HER) for any
Inter-Site Network Layer 2 BUM traffic)
MP-BGP - EVPN
Multi-Site
Orchestrator
ACI Multi-Site
Namespace Normalisation VNID  16678781
Class-ID: 49153 Translation of Class-ID, VNID
Inter-Site (scoping of name spaces)
VNID  16678781 Network
Spine Translation Table
Class-ID: 49153
Rem. Site Local Site
VNID 16678781 16547722

MP-BGP - EVPN Class-ID 49153 32770
Multi-Site VNID  16547722 EP1

Orchestrator EPG
C EP2 EPG
Class-ID: 32770
VNID  16678781
Class-ID: 49153 EP1 EP2 Site 2
Site 1 EPG C EPG
Leaf to Leaf VTEP, Class-ID is local to the Fabric
Leaf to Leaf VTEP, Class-ID is local to the Fabric
VNID Class-ID Tenant Packet
VNID Class-ID Tenant Packet VNID Class-ID Tenant Packet
• Maintain separate name spaces with ID translation performed on the spine nodes
• Requires specific HW on the spine to support for this functionality
• Multi-Site Orchestrator instructs local APIC to program translation tables on spines
ACI Multi-Site ACI 4.0(2)
Release
Preferred Groups
Contract required to
Multi-Site Preferred Group communicate with EPG(s)
external to the Preferred Group
App DB
Shared
C1
EPG
Open
Web
communication
• Multisite Preferred Group configuration directly from the Multi-Site Orchestrator is supported from
ACI 4.0(2) release
• Creates ‘shadow’ EPGs and translation table entries ‘under the hood’ to allow inter-site communication
• Typically desired in legacy to ACI migration scenarios
• vzAny support for Multi-Site scoped for a future ACI release
• ‘VRF unenforced’ option not supported with ACI Multi-Site
ACI Multi-Site Networking Options
Per Bridge Domain Behavior
Layer 3 only across sites
1
ISN
Site Site
1 2
 Bridge Domains and subnets not

extended across Sites
 Layer 3 Intra-VRF or Inter-VRF
communication (shared services
across VRFs/Tenants)
MSO GUI
(BD)
L3 Only across Sites
Why not using just normal routing across independent fabrics?
Independent ACI Fabrics (no ACI Multi-Site)
L3Out L3Out
Need to apply a contract Need to apply a contract

WAN between internal EPG and
between internal EPG and
Ext-EPG associated to the Ext-EPG associated to the
L3Out in Fabric 1 L3Out in Fabric 2
Mandates the use of a multi-VRF
capable backbone network (VRF-
Lite, MPLS-VPN, etc.) to extend
multiple VRFs across fabrics
Per Bridge Domain Behavior
Layer 3 only across sites IP Mobility without BUM flooding Layer 2 adjacency across Sites
1 2 3
ISN ISN ISN
Site Site Site Site 2
Site Site Site
1 2 1 2 1 2
 Bridge Domains and subnets not  Same IP subnet defined in separate  Interconnecting separate sites for
extended across Sites Sites fault containment and scalability
reasons
 Layer 3 Intra-VRF or Inter-VRF  Support for IP Mobility (‘cold’ and
communication (shared services ‘live’* VM migration) and intra-  Layer 2 domains stretched across
across VRFs/Tenants) subnet communication across sites Sites, support for application
clustering
 No Layer 2 BUM flooding across
sites  Layer 2 BUM flooding across
sites
MSO GUI MSO GUI MSO GUI

(BD) (BD) (BD)
*’Live’ migrationBRKACI-2125
#CLMEL officially supported from
© 2019 Cisco and/orACI release
its affiliates. 3.2
All rights reserved. Cisco Public 31
ACI Multi-Site
Continuous Scale Improvements
ACI Release 3.0 ACI Release 3.1 ACI Release 3.2 ACI Release 4.0
MSO 1.0 MSO 1.1 MSO 1.2 MSO 2.0

Number Of Sites 5 8 10 12
Max Leafs (across sites) 250 800 1200 1600
Tenants 100 200 300 400
VRF 400 400 800 1000
BD 800 2,000 3,000 4,000
EPGs 800 2,000 3,000 4,000
Contracts 1,000 2,000 3,000 4,000
L3Out (External EPGs) 500 500 500 500
Isolated EPGs N/A 400 400 400
ACI Multi-Site
CloudSec Encryption for VXLAN Traffic
Encrypted Fabric to Fabric Traffic
[GCM-AES-256-XPN (64-bit PN)])
CloudSec = “TEP-to-TEP MACSec”
VTEP IP MACSec VXLAN Tenant Packet
VTEP Information
in Clear Text
Inter-Site Network
MP-BGP - EVPN
Multi-Site
Orchestrator
Support planned for ACI 4.0 release for FX line cards and 9364C/9332C platforms
Introducing the ACI Multi-Site Orchestrator
ACI Multi-Site
Multi-Site Orchestrator (MSO)
• Micro-services architecture
 Three MSO nodes are clustered and run concurrently
(active/active)
REST
 vSphere VM only form factor initially (physical appliance
GUI
API planned for a future ACI release)
• OOB Mgmt connectivity to the APIC clusters deployed in
ACI Multi-Site Orchestrator separate sites
150 msec RTT
(max)
 Not supported to establish MSO-APIC connectivity ‘inband’
VM VM VM in the ACI fabric
 Recommended not to connect the MSO nodes to the ACI
Hypervisor fabric
• Main functions offered by MSO:
1 sec RTT
(max)  Monitoring the health-state of the different ACI Sites
 Provisioning of day-0 configuration to establish inter-site
EVPN control plane
…..  Defining and provisioning policies across sites
Site 1 Site 2 Site n
 Day-2 operation functionalities
ACI Multi-Site
MSO Dashboard
• Health/Faults for all managed sites

• Easily way to identify stretched policies across sites
• Quickly search for any deployed inter-site policy
• Provide direct access to the APIC GUIs in different sites
ACI Multi-Site
MSO Schema and Templates
 Template = APIC policy definition

Schema
POLICY
(App and Network) TemplateDEFINITION
Template
 Template is the scope/granularity EP1
EPG
C EP2
EPG
of the policies that can be pushed to

one (or more) sites
SITE
 Schema = container of Templates LOCAL
sharing a common use-case

• As an example, a schema can be
dedicated to a Tenant Site 1 Site 2
 Scope of change: policies in different EFFECTIVE
POLICY
EFFECTIVE
POLICY
templates can be pushed to separate
sites at different times
ACI Multi-Site
UCSD and Ansible Integration
Shipping ACI 4.1/MSO 2.1
UCSD 6.6 and Ansible Main Functions
UCSD 6.6 Site Management
Ansible Site Infra config and test connectivity
Orchestration
MSC site inventory
APIC site management (cross-launch)
User Management
Tenant Lifecycle and Site Association
Schema and Template lifecycle (AP, EPGs, Contracts, VRF, BD, etc … )
L3Out and External EPG
Deploy Tenants and Schemas to sites
Monitoring MSC and Management
….. Import brownfield tenant policies and deploy across sites
Site 1 Site 2 Site n
Trouble-shooting
Inter-Site Connectivity Deployment
Considerations
ACI Multi-Site
Inter-Site Network (ISN) Requirements
Inter-Site Network
MP-BGP - EVPN
Multi-Site
Orchestrator
• Not managed by APIC, must be independently configured (day-0 configuration)

• IP topology can be arbitrary, not mandatory to connect to all spine nodes
• Main requirements:
 OSPF on the first hop routers to peer with the spine nodes and exchange TEP address reachability
 Must use sub-interfaces (with VLAN tag 4) toward the spines
 Increased end-to-end MTU support (at least 1550 B)
ACI Multi-Site and MTU
Different MTU Meanings
1. Data Plane MTU: MTU of the traffic

generate by endpoints (servers,
routers, service nodes, etc.) 2 ISN
connected to ACI leaf nodes MP-BGP EVPN
• Need to account for 50B of

overhead (VXLAN encapsulation) for
inter-site communication
2. Control Plane MTU: for CPU Multi-Site
Orchestrator
generated traffic like EVPN across 1 1
sites
• The default value is 9000B, can be
tuned to the maximum MTU value
supported in the ISN
ACI Multi-Site and MTU
Tuning MTU for EVPN Traffic across Sites
Configurable MTU
ISN
MP-BGP - EVPN
Multi-Site
Orchestrator
 Control Plane MTU can be set leveraging the

“CP MTU Policy” on APIC Modify the default
9000B MTU value
 The required MTU in the ISN would then
depend on this setting and on the Data Plane
MTU configuration
Always need to consider the VXLAN encapsulation
overhead for data plane traffic (50/54 bytes)
Control and Data Plane
ACI Multi-Site
BGP Inter-Site Peers
• Spines connected to the Inter-Site Network
perform two main functions:
Inter-Site
Network 1. Establishment of MP-BGP EVPN peerings with
Anycast VTEP Addresses: spines in remote sites
O-UTEP & O-MTEP
 One dedicated Control Plane address (EVPN-RID) is
assigned to each spine running MP-BGP EVPN
2. Forwarding of inter-sites data-plane traffic
EVPN-RID 4  Anycast Overlay Unicast TEP (O-UTEP): assigned to
all the spines connected to the ISN and used to
EVPN-RID 1 source and receive L2/L3 unicast traffic
EVPN-RID 2 EVPN-RID 3
 Anycast Overlay Multicast TEP (O-MTEP): assigned
to all the spines connected to the ISN and used to
receive L2 BUM traffic
• EVPN-RID, O-UTEP and O-MTEP addresses

are assigned from the Multi-Site Orchestrator
and must be routable across the ISN
ACI Multi-Site
Exchanging TEP Information across Sites IP Network Routing Table
O-UTEP A, O-MTEP A
EVPN-RID S1-S4
O-UTEP B, O-MTEP B
Filter out the
EVPN-RID S5-S8
advertisement of internal
• OSPF peering between spines and TEP pool into the ISN
Inter-Site network Inter-Site

• Exchange of External Spine TEP OSPF Network OSPF
addresses (EVPN-RID, O-UTEP and
O-MTEP) across sites S1 S2 S3 S4 S5 S6 S7 S8
IS-IS to OSPF
Internal TEP Pool information not needed mutual redistribution
to establish inter-site communication TEP Pool 1 TEP Pool 2
(should be filtered out on the first-hop
ISN router)
Multi-Site
In any case, best practice is to avoid Orchestrator
using overlapping TEP Pools across sites Site 1 Site 2
Leaf Routing Table Leaf Routing Table

IP Prefix Next-Hop IP Prefix Next-Hop
O-UTEP B Pod1-S1, Pod1-S2, O-UTEP A Pod2-S1, Pod2-S2,
Pod1-S3, Pod1-S4 Pod2-S3, Pod2-S4
ACI Multi-Site
Inter-Site MP-BGP EVPN Control Plane
S3-S4 Table S5-S8 Table

• MP-BGP EVPN used to
EP1 Leaf 1 Leaf 4
communicate Endpoint (EP) MP-BGP EVPN
EP2
O-UTEP B
information across Sites EP2 EP1 O-UTEP A
MP-iBGP or MP-EBGP peering

options supported Inter-Site
Network
Remote host route entries (EVPN
O-UTEP A O-UTEP B
Type-2) are associated to the remote
site Anycast O-UTEP address S1 S2 S3 S4 S5 S6 S7 S8
Multi-Site
• Automatic filtering of endpoint COOP
Orchestrator
COOP
information across Sites
Host routes are exchanged across
EP2
sites only if there is a cross-site EP1
Site 1 Site 2
contract requiring communication
between endpoints Define and push inter-site policy
EP1 EP2
EPG
C EPG
Connecting to the External Layer 3 Domain
‘Traditional’ L3Outs on the BL Nodes
Client
L3Out WAN
• Connecting to WAN Edge devices at

Border Leaf nodes
• VRF-Lite hand-off for extending L3 multi-
tenancy outside the ACI fabric
Border Leafs • Up to 400 L3Outs/VRFs currently supported on
the same BL nodes pair
• Support for host routes advertisement out
of the ACI Fabric from ACI release 4.0(1)
‘GOLF’ L3Outs
= VXLAN Encap/Decap
Different WAN
Hand-Off options:
VRF-Lite, MPLS-
VPN, LISP*
Client
WAN
OTV/VPLS
• Connecting to WAN Edge devices at Spine
GOLF Routers
(ASR 9000, ASR 1000,
nodes (directly or indirectly)
Nexus 7000)  VXLAN data plane with MP-BGP EVPN control
plane
• High scale tenant L3Out support
• Simplified and automated tenant L3Out
configuration with OpFlex
• Support for host routes advertisement out
of the ACI Fabric
Multi-Site and L3Outs
Endpoints always Use Local L3Outs for Outbound Traffic
Supported Design
✓ Not Supported Design
❌
Inter-Site Network Inter-Site Network
X
L3Out L3Out L3Out

Site 1 Site 2 Site 1
WAN WAN
Note: the same consideration applies to both Border Leaf L3Outs and GOLF L3Outs
ACI 4.2(1)
ACI Multi-Site and L3Outs Release
Support of Inter-Site L3Out
• Starting with ACI Release 4.2(1) it will be

Inter-Site Network possible for endpoints in a site to send traffic to
resources (WAN, Mainframes, etc.) accessible
via L3Out of a remote site
• Traffic will be directly encapsulated to the TEP
of the remote BL nodes
x
• The BL nodes will get assigned an address part of
L3Out
Site 1
L3Out
Site 2
an additional (configurable) prefix that must be
routable across the ISN
WAN
• Same solution will also support transit routing
across sites (L3Out to L3Out)
Multi-Site and L3Out
Endpoints always Use Local L3Outs for Outbound Traffic
Inter-Site Network
Site 1 Site 2
Web-EPG C1 Ext-EPG
Traffic dropped
because of lack of
state in the FW
IP Subnet
10.10.10.10 10.10.10.0/24 Active/Standby IP Subnet 10.10.10.11
10.10.10.0/24
Active/Standby
Multi-Site and L3Out ACI 4.0(1)
Release
Use of Host-Routes Advertisement
Inter-Site Network
Site 1 Site 2
Web-EPG C1 Ext-EPG
L3Out L3Out
Site 1 Site 2
10.10.10.10 Host routes 10.10.10.11
10.10.10.10/32 Active/Standby Active/Standby Host routes
10.10.10.11/32
*Alternative could be
running an overlay solution
Host-routes
(LISP, GRE, etc.) injected into the
WAN* Enabled at
the BD level
• Ingress optimisation requires host-routes advertisement on the L3Out
 Native support on ACI Border Leaf nodes available from ACI release 4.0
 Supported also on GOLF L3Outs #CLMEL
BRKACI-2125 BRKACI-2125 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Multi-Site and Network Services
Integration
Multi-Site and Network Services Deployment options fully
supported with ACI Multi-Pod
Integration Models
ISN
• Active and Standby pair deployed across Pods

• Currently supported only if the FW is in L2 mode or in
L3 mode but acting as default gateway for the
endpoints
Active Standby
Active/Active FW cluster nodes stretched across Sites

ISN
•
(single logical FW)
• Requires the ability of discovering the same MAC/IP info
in separate sites at the same time
Active/Active Cluster
• Not currently supported (scoped for a future ACI release)
ISN • Recommended deployment model for ACI Multi-Site

• Option 1: supported from 3.0 for N-S if the FW is
connected in L3 mode to the fabric  mandates the
deployment of traffic ingress optimisation
• Option 2: supported from 3.2 release with the use of
Active/Standby Active/Standby #CLMEL Service Graph with
BRKACI-2125 © 2019 Policy
Cisco and/orBased Redirection
its affiliates. All (PBR)
rights reserved. Cisco Public 70
Use of Service Graph and Policy Based Redirection
North-South Communication – Inbound Traffic
Inter Site
Network
Site1 Site2
Compute leaf
always applies
the PBR policy Compute leaf
always applies
EPG EPG the PBR policy
Ext C Web
Consumer Provider
(Provider) (Consumer)
L3Out-Site1 L3Out-Site2
10.10.10.10 10.10.10.11
L3 Mode L3 Mode
Active/Standby Active/Standby
• Inbound traffic can enter any site when destined to a stretched subnet (if ingress optimisation is not deployed or
possible)
• PBR policy must always be applied on the compute leaf node where the destination endpoint is connected
 Requires the VRF to have the default policies for enforcement preference and direction
 Supported only intra-VRF in ACI release 3.2
 Ext-EPG and Web EPG can indifferently be provider or consumer of the contract
North-South Communication – Outbound Traffic
Inter Site
Network
Site1 Site2
Compute leaf
always applies
the PBR policy Compute leaf
always applies
EPG EPG the PBR policy
Ext C Web
Consumer Provider
(Provider) (Consumer)
L3Out-Site1 L3Out-Site2
10.10.10.10 10.10.10.11
L3 Mode L3 Mode
• PBR policy always applied on the same leaf where it was applied for inbound traffic
• Ensures the same service node is selected for both legs of the flow
• Different L3Outs can be used for inbound and outbound directions of the same flow
East-West Communication (1)
Inter Site
Network
Site1 Site2
Consumer leaf
always applies
the PBR policy
EPG EPG
Web C App
Consumer Provider
L3 Mode L3 Mode EPG

EPG
Web App
*Changed to Provider side

 EPGs can be locally defined or stretched across sites enforcement from ACI release
 EPGs can be part of the same VRF or in different VRFs (and/or Tenants) 4.0
 PBR policy is always applied on the leaf switch where the Consumer* endpoint is connected
• The Consumer leaf always redirects traffic to a local service node
East-West Communication (2)
Inter Site
Network
Site1 Site2
Consumer leaf
always applies Provider leaf
the PBR policy does not apply
EPG EPG
Web C App the PBR policy
Consumer Provider
L3 Mode L3 Mode EPG

EPG
Web App
 The Provider leaf must not apply PBR policy to ensure proper traffic stitching to the FW
node that has built connection state
 Ensures both legs of the flow are handled by the same service node
Multi-Site and Virtual Machine Manager
(VMM) Integration
ACI Multi-Site and VMM Integration
Option 1 – Separate VMM per Site
ISN
Site 1 VMM Domain VMM Domain Site 2

DC1 DC2
VMM 1 VMM 2
HV vSwitch1
HV HV Managed HV vSwitch2
HV HV
by VMM 1
Managed
by VMM 2
• Typical deployment model for an ACI Multi-Site

• Creation of separate VMM domains in each site, which are then exposed to
the Multi-Site Orchestrator
ACI Multi-Site and VMM Integration
Workload Migration across Sites
ISN
Site 1 VMM Domain VMM Domain Site 2

DC1 DC2
vCenter vCenter
Server 1 Server 2
SRM SRM
HV HVVDS1 HV
EPG1 HV HVVDS2 HV
EPG1
Live vMotion/Cold Migration

• Live virtual machines migration across sites is supported only with vCenter deployments
(both for single or multiple vCenter options)
 Requires vSphere 6.0 and newer, no support for DRS, vSphere HA/FT
• Use of Site Recovery Manager (SRM) or similar higher level orchestrator for workload
recovery across sites
Multi-Pod and Multi-Site Integration
ACI Multi-Pod and Multi-Site ACI 3.2(1)
Release
Main Use Cases
 Adding a Multi-Pod Fabric as a ‘Site’ on the Multi-Site Orchestrator (MSO)
 Converting a single Pod Fabric (already added to MSO) to a Multi-Pod fabric
ACI Multi-Pod and Multi-Site
Connectivity between Pods and Sites
Single external network used
for IPN and ISN
IPN/ISN
1st Gen 1st Gen
APIC Cluster
Pod ‘A’ Pod ‘B’
Site 1 Site 2
 Only 2nd generation spines must be connected to the external network

• Need to add 2nd gen spines in each Pod (at least two per Pod) and migrate connections to the IPN from 1st gen
spines to 2nd gen spines
 Single ‘infra’ L3Out and set of uplinks to carry both Multi-Pod and Multi-Site East-West traffic
IP WAN
Separate networks
used for IPN and ISN
IPN
Site 2
1st Gen 1st Gen
APIC Cluster
Site 1 Site 2

Not Supported Topology
Separate uplinks
between spines IP WAN
and external
networks
IPN
Site 2
1st Gen 1st Gen
APIC Cluster
Site 1 Site 2

BGP Spine Roles
 Spines in each Multi-Pod fabric can have one

of those two roles:
1. BGP Speakers: establish EVPN peerings with BGP
IPN/ISN speakers in remote sites and with BGP Forwarders
in the local site (intra- and inter- Pods)
BGP Forwarders
BGP Forwarders
BGP
BGP Forwarders
Forwarders
• Recommended to deploy two speakers per
BGP Speakers
BGP Speakers Multi-Pod fabric (in separate Pods)
1st Gen 1st Gen

• Explicitly configured on MSO
• Multi-Site Speaker must be a Multi-Pod spine
as well
2. BGP Forwarders: establish BGP EVPN peerings
with BGP speakers in the local site
APIC Cluster
Pod ‘B’
• All the spines that are not speakers implicitly
become forwarders
Site 1
Inter-Site and Intra-Site EVPN Sessions
= Inter-Site MP-BGP EVPN Peering (Speaker-to-Speaker)
= Intra-Site MP-BGP EVPN Peering (Speaker-to-Forwarders)
Site 2
IP
BGP BGP
Speaker Speaker
IP WAN
IPN
BGP Forwarder Forwarder Forwarder Forwarder BGP

Speaker Speaker
Site 1
TEP Pools Deployment and Advertisement
Filter out the
advertisement of
TEP Pool advertisement
10.1.0.0/16
the TEP Pool into not needed for inter-Sites
Site 2
IP the backbone
communication
IP WAN
Filter out the
All Pods TEP Pools
TEP Pool Site 2 advertisement of
advertised into the
10.1.0.0/16 the TEP Pools into
IPN
the backbone
IPN
10.1.0.0/16
TEP Pool Pod1 10.2.0.0/16 10.3.0.0/16 TEP Pool Pod3
10.1.0.0/16 TEP Pool Pod2 10.3.0.0/16
10.2.0.0/16
Site 1
Migration Scenarios
ACI Multi-Site
Migration Scenarios
Site 1
2
Site 1
1
Site 2 Site 2
2 2 1 3
Site 1 Site 2 Site 1 Site 2

Green Field Green Field Brown Field Green Field
1. Model new tenant and inter-site 1. Import existing tenant policies from site 1 to
policies on the ACI Multi-Site a new template on ACI Multi-Site
Orchestrator and associate the Orchestrator
template to the sites
2. Associate the template also to site 2
2. Push policies to the ACI sites
3. Push template policies to site 2
ACI Multi-Site
Migration Scenarios Import Policies
from Site 1 and
Push it to Site 2
Fabric 1 ISN
‘Brownfield’ ACI Fabric Site 1 Site 2
to Multi-Site
ISN
Pod ‘A’ IPN Pod ‘B’ IPN
Multi-Pod to Pod ‘A’ Pod ‘B’
‘Hierarchical Multi-Site’ Site 2
APIC Cluster
APIC Cluster
Multi-Pod From ACI 3.2(1) release Site 1
Fabric 1 Fabric 2 ISN

Multi-Fabric Design to
Site 1 Site 2
Multi-Site
Inter-Site
App
L2/L3
DCI Scoped for the future
#CLMEL 95
Multi-Fabric BRKACI-2125 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Conclusions
Multi-Pod and Multi-Site
Complementary Architectures
Multi-Pod Fabric ‘A’ (AZ 1)
Pod ‘1.A’ Pod ‘2.A’
ACI Multi-Site
Multi-Pod Fabric ‘B’ (AZ 2)
Application
Pod ‘1.B’
workloads
Pod ‘2.B’
deployed across
availability zones #CLMEL BRKACI-2125 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Where to Go for More Information
 ACI Multi-Pod White Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737855.html?cachemode=refresh
 ACI Multi-Pod Configuration Paper

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739714.html
 ACI Multi-Pod and Service Node Integration White Paper

 BRKACI-2003 @ Cisco Live Barcelona 2019

https://ciscolive.cisco.com/on-demand-library/?search=BRKACI-2003#/session/1532112828758001tmf6
 ACI Multi-Site White Paper

 Deploying ACI Multi-Site from Scratch

https://www.youtube.com/watch?v=HJJ8lznodN0
Q&A
#CLMEL
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Complete Your Online Session Evaluation
• Give us your feedback and receive a

complimentary Cisco Live 2019 Power
Bank after completing the overall event
evaluation and 5 session evaluations.
• All evaluations can be completed via
the Cisco Live Melbourne Mobile App.
• Don’t forget: Cisco Live sessions will be
available for viewing on demand after
the event at:
https://ciscolive.cisco.com/on-demand-library/
#CLMEL BRKACI-2125 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you
#CLMEL
#CLMEL

ACI Multisite Deployment - BRKACI-3502

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ACI Multisite Deployment - BRKACI-3502

Uploaded by

Copyright:

Available Formats

#CLMEL

Max Ardica – Principal Engineer (DCNBU)

At the end of the session, the participants should be able to:

• ACI Multi-Pod Quick Review

• ACI Multi-Site Deep Dive

• Conclusions and Q&A

5,800+ 50+% 65+

ACI Single Pod Fabric ACI Multi-Site ACI Multi-Cloud

… ACI 3.1/4.0 - Remote

ACI 1.0 - ACI 3.0 – Multiple ACI 4.1 – ACI Extensions

• Regions – Separate large geographical areas, each Amazon Web Services

Fabric ‘A’ (AZ 1)

Fabric ‘B’ (AZ 2)

Multi-Pod Fabric ‘A’ (AZ 1)

Pod ‘1.A’ Pod ‘2.A’

Multi-Pod Fabric ‘B’ (AZ 2)

Pod ‘1.B’Application Pod ‘2.B’

Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain

 Need to scale up a single ACI fabric above

 True Active/Active DC deployments

• Scale-up model to build a very • Data Centre Interconnect (DCI)

• ACI Multi-Site introduced from release 3.0(1)

• New 9364C/9332C non modular spine

Network information carried across Identity information carried across

VTEP IP VNID Class-ID Tenant Packet No Multicast Requirement

VNID 16678781 16547722

Multi-Site VNID  16547722 EP1

 Bridge Domains and subnets not

Why not using just normal routing across independent fabrics?

Independent ACI Fabrics (no ACI Multi-Site)

Need to apply a contract Need to apply a contract

MSO GUI MSO GUI MSO GUI

MSO 1.0 MSO 1.1 MSO 1.2 MSO 2.0

Max Leafs (across sites) 250 800 1200 1600

Tenants 100 200 300 400

VRF 400 400 800 1000

BD 800 2,000 3,000 4,000

EPGs 800 2,000 3,000 4,000

Contracts 1,000 2,000 3,000 4,000

L3Out (External EPGs) 500 500 500 500

Isolated EPGs N/A 400 400 400

VTEP IP MACSec VXLAN Tenant Packet

• Health/Faults for all managed sites

 Template = APIC policy definition

of the policies that can be pushed to

sharing a common use-case

• Not managed by APIC, must be independently configured (day-0 configuration)

1. Data Plane MTU: MTU of the traffic

• Need to account for 50B of

generated traffic like EVPN across 1 1

 Control Plane MTU can be set leveraging the

• EVPN-RID, O-UTEP and O-MTEP addresses

Inter-Site network Inter-Site

Leaf Routing Table Leaf Routing Table

S3-S4 Table S5-S8 Table

MP-iBGP or MP-EBGP peering

• Connecting to WAN Edge devices at

L3Out L3Out L3Out

• Starting with ACI Release 4.2(1) it will be

• Active and Standby pair deployed across Pods

Active/Active FW cluster nodes stretched across Sites

ISN • Recommended deployment model for ACI Multi-Site

L3 Mode L3 Mode EPG

*Changed to Provider side