ACI_Acceleration_Advanced_V1.0

ACI Acceleration Series
Part 2: Deep Dive into ACI Anywhere
Huyen Duong, TME Technical Leader - CNBU

Jayesh Singh, TME - CNBU
Daheng Yang, TME Technical Leader - CNBU
Sep 2022
Part 2: Deep Dive into ACI Anywhere
ACI Beyond The Single Data Center
Modules
1 ACI Recap
5 Nexus Dashboard Orchestrator
Your central place for connectivity & policy control
2 Introduction to ACI Anywhere

Extend ACI to Many Locations 6 ACI Multi-Site
Scale to Multiple Locations, Sites and Clouds
3 ACI Multi-Pod
Extending the Metro Area 7 L4-L7 Services and Other Considerations
ACI Remote Leaf

4 Addressing smaller locations
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public
Day 1 Module 1
ACI Acceleration Series ACI Recap

Deep Dive into ACI Anywhere
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco

CiscoPublic
Confidential
ACI Policy Model
ACI Recap… Tenant
L3Out
VRF-1
EPG 10 c EPG 20 c EPG 30

Bridge Domain Bridge Domain Bridge Domain
10.1.1.1/24 20.1.1.1/24 30.1.1.1/24
App Profile
Why ACI?
ACI Fabric
• Centralized Management
• Policy-based Segmentation
• Built-in Multi-Tenancy
• Zero Trust Architecture
• Automated Fabric Bring up
• Anycast Gateway
• Seamless Workload Mobility Integrated VXLAN Overlay

• VM & Container Visibility
Nexus 9000 Series

L3 Out Basic Functions
Review: Understanding ACI Discovery, Protocols and Automation
APICs
Fully Automated
BGP RR SPINES
MP-BGP
IS-IS
VXLAN
Overlay-1 VRF
Route LEAVES
Route Redistribution
Advertise internal VXLAN Encap/Decap
Redistribution BD Subnets routesinto
externally
ACI
(via border Leaf)
Physical Network
External Routes Access Policies - L3 Domain
67.0.0.0/8 Logical Network
68.0.0.0/8
L3 Out (OSPF/BGP/EIGRP peering, static route via a specific port)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential L3 Out External EPG (prefix classification)
What are the differences?
Review
L3Out
IP and Mac Address associated EPG EPG
External L3 Prefixes associated
& redistributed inside the ACI fabric
OSPF EIGRP BGP
0.0.0.0/0
200.100.0.0/16
15.10.24.0/24
Associate End-Points Associate Prefixes

learned through an L3 Out
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
L4-L7 services
Using Service-Graphs L4-L7
Bare-metal
BD: Web_BD EPG EPG BD: DB_BD

GW - Subnet: 1.1.1.1/24 GW - Subnet: 2.2.2.1/24
Web Contract
Redirect only
DB
HTTP to FW
(PBR)
Define once, use anywhere
DB
ICMP HTTP 1.1.100.100 2.2.200.100

FW_Outside_BD FW_Inside_BD
Firewall
Web Server 1 Web Server 2 (2-arm) DB Server 1
1.1.1.2 1.1.1.22 Virtual | Physical 2.2.2.3
* ACI 4.2 and later has ARP flooding enabled by default for every BD. ARP Flooding should be enabled for previous versions
Day 1 Module 2
ACI Acceleration Series Introduction to ACI

Deep Dive into ACI Anywhere Anywhere

CiscoPublic
Confidential
A Data Center Needs to go Anywhere the Data Is
IoT Edge
Enterprise DC 5G Telco Edge
Public Cloud / IaaS Enterprise Edge
Private Cloud Colo / Bare Metal Cloud
Introducing
ACI Anywhere
Cisco ACI Anywhere
Any workload, any location, any cloud
ACI Anywhere
Remote Leaf / ACI Mini APIC Single Site / Multi- Multi-Cloud Extensions
Pod / Multi-Site
IP IP
WAN WAN
Remote Location On Premises Public Cloud
Security Everywhere Analytics Everywhere Policy Everywhere
ACI Anywhere: Extend the simplicity
Multiple locations managed from a single point
ACI 6.x
ACI 5.2
ACI 4.2
Remote-Leaf Cloud ACI

Multi-Pod ACI 3.0 2017 ACI 3.2 Extension
2016
2019 Ongoing Feature
Enhancements
ACI 2.0 ACI 3.1 ACI 4.1 Today
Multi-Site Multi-Pod
2017 controlled by
Multi-Site
2018
Day 1 Module 3
ACI Acceleration Series ACI Multi-Pod


CiscoPublic
Confidential
An ACI Site is represented by an APIC Cluster
Single Site ACI Multi-Site ACI
Nexus Dashboard
Orchestrator
Separate management and Single management and logical network

configuration per site (isolated sites) configuration with automated interconnect
You can run ACI anywhere your business takes you
An ACI Site is represented by an APIC Cluster
Single Site ACI
Single Pod Multi-Room Data Centers

Metropolitan Area DCs
Multi-Pod
Remote-Leaf Pod ~= Availability Zone
Separate management and

configuration per site (isolated sites)
You can run ACI anywhere your business takes you
ACI Multi-Pod
Any Routed Network
(IPN)
Other Rooms/Data Centers
Active-Active Data Centers
Pod N
Single Management Domain
Pod 1 Pod 2
(All Pods)
High level of control needed for Short distances (50ms RTT required) Single APIC cluster
IPN: Multicast and MTU Scales up to 12 Pods / 500 total leafs Automated L2 DCI VXLAN extension
Supported ACI Multi-Pod topologies
Intra-data center
10G*/40G/100G/400G 10G*/40G/100G/400G
Pod 1 Pod n
APIC Cluster
* 10G only with QSA adapters on EX spines
Two data center sites directly connected
(up to 50 msec RTT)

10G*/40G/100G/400G 10G*/40G/100G/400G
Pod 1 Pod 2
APIC Cluster
Three data center sites directly connected
(up to 50 msec RTT)

10G*/40G/100G/400G 10G*/40G/100G/400G
Pod 1 Pod 2
10G*/40G/100G/400G
Pod 3
Multiple pods interconnected by a generic L3 network
10G*/40G/100G/400G 10G*/40G/100G/400G
MPLS or SP Cloud
10G*/40G/100G/400G
10G*/40G/100G/400G (up to 50 msec RTT)
Any Routed
Network (IPN)
Multi-Pod ACI Pod 2
On-Prem
(with IPN)
IPN First Hop
Underlay OSPF or eBGP (VLAN

4)
Access-Policies
E1/1.4 E1/1.4 L3 Domain
IP Address IP Address
VLAN Pool
AEP
Policy Group
Managed and configured Spine Profiles
automatically by APIC Spine Nodes
L3 Out (infra)
Fab Ext Conn Policy
Fab Ext Conn Profile
No need to connect all spines to IPN
Pod 1
Any Routed
Network (IPN)
Multi-Pod ACI Pod 2
On-Prem
(with IPN)
IPN First Hop Not configured nor

managed by APIC
E1/1.4 E1/1.4

4)
E1/1.4 E1/1.4 L3 Out advertises:
Spine-IPN IP addresses
VTEPs
Managed and configured
APIC internal IPs
automatically by APIC Spine Nodes (from TEP Pool)
Loopbacks

Pod 1
Any Routed
Network (IPN)
Multi-Pod ACI
On-Prem
(with IPN)
IPN First Hop

E1/1.4 E1/1.4 E1/1.4 E1/1.4
IP Address IP Address IP Address IP Address

4)
OSPF or eBGP (VLAN
4)
E1/1.4 E1/1.4
DHCP Request
automatically by APIC Spine Nodes Spine Nodes
Unconfigured Pod 1 Pod 2
Any Routed
Network (IPN)
Multi-Pod ACI Pod 2
On-Prem
(with IPN)
IPN First Hop Not configured nor

managed by APIC
E1/1.4 E1/1.4

4)
E1/1.4 E1/1.4 L3 Out advertises:
Spine-IPN IP addresses
VTEPs
APIC internal IPs
automatically by APIC Spine Nodes (from TEP Pool)
Loopbacks

Pod 1
Any Routed
Network (IPN)
Multi-Pod ACI
On-Prem DHCP Relay to APICs
(with IPN)
IPN First Hop

E1/1.4 E1/1.4 E1/1.4 E1/1.4

4)
E1/1.4 E1/1.4

Internal IPs:
10.0.0.1 | 10.0.0.2 | 10.0.0.3
Pod 1 Pod 2
Any Routed
Network (IPN)
Multi-Pod ACI
On-Prem
(with IPN)
IPN First Hop

E1/1.4 E1/1.4 E1/1.4 E1/1.4

4)
OSPF or eBGP (VLAN
4)
Automatically deploy
E1/1.4 E1/1.4
IP Address IP Address configuration

A TEP Pool is needed for each Pod

Pod 1 Pod 2
Pod 2
New nodes
Spine 103:discovered!
10.1.0.10
Spine 104: 10.1.0.11 TEP Pool Pod 1: 10.0.0.0/16
TEP Pool Pod 2: 10.1.0.0/16
Any Routed
Network (IPN)
Multi-Pod ACI
On-Prem
(with IPN)
IPN First Hop
Underlay
MP-BGP EVPN VXLAN

Same L2 Subnets are stretched across pods
Anycast gateway (BD Subnets) distributed across pods Pod 1 Pod 2

MP-BGP is used as the Control Plane between Pods
L2 Extension
MP-BGP EVPN VXLAN
Multi-Pod ACI Include the following blocks in RP:

Enable PIM sparse on every interface
225.0.0.0/8 (GIPo) and 239.255.255.240
On-Prem IPN Set a Rendevous-Point (RP)
(ARP Gleaning)
(with IPN)
Multicast
Not in COOP!
Flood on GIPo or ARP
Gleaning
Spine
Nodes
Where is 1.1.1.3?
It’s me! Store my EP
Flood/HW Proxy
info in your EP Tables
1.1.1.2 1.1.1.3
PIM BI-Dir is needed in MultiPod to extend Bridge Domain GIPo

Pod 1 Pod 2
Consider 50+ byte payload increase in IPN MTU (VXLAN)
Any Routed
Network (IPN)
Multi-Pod ACI
On-Prem
(with IPN)
IPN First Hop
Underlay
MP-BGP EVPN VXLAN

You can distribute APICs across pods

Pod 1 Pod 2
Recommended: Add a backup APIC (quorum)
Up to 500 leaf nodes supported in a Multi-Pod Site
L2 Extension
Multi-Pod ACI
On-Prem IPN
(with IPN)
EVPN VXLAN
Spine
Nodes
E E
Objectives
1.1.1.2 1.1.1.3
Move workloads across cities with no interruption Pod 1 Pod 2
Preserve the same EPG and policies across pods
Multi-Pod ACI
1 Setup IPN
2 Run the Multipod Wizard
3 Test connectivity and verify network extension
33
Multi-Pod ACI
1 Setup IPN E1/53.4 E1/52.4

N9K-IPN-Site-1
NX-OS Manual
Configuration
E1/54.4
E1/51.4
E1/61 E1/61
E1/31 E1/31
Spine Spine Spine Spine

101 102 103 104
Pod 1 Pod 2
34
Multi-Pod ACI
PIM ospf dhcp Interface-vlan
OSPF area 0 PIM sparse mode MTU PIM RP
1 Setup IPN E1/53.4 E1/52.4

N9K-IPN-Site-1
NX-OS Manual
Configuration
.2 E1/54.4 VRF: msite
DHCP Relay
E1/51.4
172.16.113.0/24
172.16.112.0/24
172.16.111.0/24
172.16.114.0/24
E1/61 E1/61
.1 E1/31 .1 E1/31
APICs Internal IPs:
10.1.0.1
10.1.0.2
10.1.0.3
101 102 103 104
Pod 1 Pod 2
35
Multi-Pod ACI
1 Setup IPN
2 Run the Multipod Wizard NEXT à
3 Test connectivity and verify network extension
36
Multi-Pod ACI
N9K-IPN-Site-1
E1/53.4 E1/52.4
2
NX-OS Manual
Run the Multipod Wizard Configuration
E1/54.4
E1/51.4
E1/61 E1/61
E1/31 E1/31

101 102 103 104
Pod 1 Pod 2
Leaf Leaf Leaf
201 202 203
37
Multi-Pod ACI
N9K-IPN-Site-1
E1/53.4 E1/52.4
2
NX-OS Manual
E1/54.4
E1/51.4
MP-BGP EVPN VXLAN
Twice OSPF OSPF
E1/61 E1/61
E1/31 E1/31
Configure Pod 1 Spine Interfaces

101 102 103 104
Configure Pod 2 Spine Interfaces
Pod 1 Pod 2
Leaf Leaf Leaf
201 202 203
38
Multi-Pod ACI
N9K-IPN-Site-1
NX-OS Manual
External TEP Pools Configuration
Used to dynamically provision:
Router-ID (1 per Spine)
Anycast TEP (1 per Pod)

101 102 103 104
Pod 1 Pod 2
41
Multi-Pod ACI
N9K-IPN-Site-1
External TEP Pools NX-OS Manual
Configuration
Pod 1: 172.16.100.0/24
Pod 2: 172.16.200.0/24
Router-ID (1 per Spine) BGP Peers

EVPN VXLAN

101 102 103 104
Pod 1 Pod 2
42
Multi-Pod ACI
N9K-IPN-Site-1
Configuration
Pod 1: 172.16.100.0/24
Pod 2: 172.16.200.0/24

1.1.1.0/24 via Anycast TEP A
Anycast TEP A Anycast TEP B

101 102 103 104
Pod 1 Pod 2
Leaf Leaf Leaf
201 202 203
1.1.1.0/24
43
Multi-Pod ACI
N9K-IPN-Site-1
E1/53.4 E1/52.4
2
NX-OS Manual
E1/54.4
E1/51.4
E1/61 E1/61
E1/31 E1/31

101 102 103 104
Pod 1 Pod 2
Leaf Leaf Leaf
201 202 203
44
Multi-Pod ACI
N9K-IPN-Site-1
E1/54.4
2
E1/52.4
172.16.112.0/24 OSPF 172.16.114.0/24
Area 0
Type P2P
E1/31 E1/31
Spine Spine
103 104
Leaf
203
Pod 2
Multi-Pod ACI
Pod 2 Nodes Discovery E1/53.4 E1/52.4

N9K-IPN-Site-1
NX-OS Manual
Configuration
DHCP Relay
E1/51.4
172.16.113.0/24
172.16.112.0/24
172.16.111.0/24
172.16.114.0/24
E1/61 E1/61
.1 E1/31 .1 E1/31
APICs Internal IPs:
DHCP Requests
10.1.0.1
10.1.0.2
10.1.0.3
101 102 103 104
New nodes discovered!

Pod 1 Pod 2
46
Multi-Pod ACI
2 Internal TEP Pool Pod 2 E1/53.4 E1/52.4

N9K-IPN-Site-1
NX-OS Manual
Configuration
DHCP Relay
E1/51.4
172.16.113.0/24
172.16.112.0/24
172.16.111.0/24
172.16.114.0/24
E1/61 E1/61
.1 E1/31 .1 E1/31
APICs Internal IPs:
10.1.0.1
10.1.0.2
10.1.0.3
101 102 103 104
Pod 1 Pod 2
47
Multi-Pod ACI
N9K-IPN-Site-1
Configuration
Pod 1 - 172.16.100.0/24
Pod 2 – 172.16.200.0/24
BGP Peers
EVPN VXLAN

Anycast TEP A Anycast TEP B

101 102 103 104
Pod 1 Pod 2
Leaf Leaf Leaf
201 202 203
48
Multi-Pod ACI
1 Setup IPN
2 Run the Multipod Wizard
3 Test connectivity
49
Multi-Pod ACI
On-Prem
(with IPN)
L2VXLAN
Extension
L3
Pod 1 Pod 2
Our Objective
Back-to-Back
2 pods maximum
Multi-Pod ACI
On-Prem
ACI 5.2(3)+
MP-BGP EVPN VXLAN
OSPF
Spine
Leaf
Pod 1 Pod 2
Automate Data Center Interconnect. Provision once, deploy anywhere
Back-to-Back With Inter-Pod Network (IPN)
2 pods maximum 12 pods maximum
Multi-Pod ACI
On-Prem
ACI 5.2(3)+
MP-BGP EVPN VXLAN MP-BGP EVPN VXLAN
OSPF or BGP OSPF or BGP

OSPF IPN
Spine Spine
Leaf Leaf
Pod 1 Pod 2 Pod 1 Pod N
Summary
Automated L2 Extension over L3 networks

ACI Multi-Pod Centralized management
1 2
Extend | Migrate Consistent & Centralized
• Active-Active Data Centers • Same policy across all pods
• Active-Standby Data Centers • Reuse configurations on all pods
• Data Center Availability Zones • Back-to-Back or IPN options
• Connect different rooms • Up to 12 Pods
• RTT < 50 ms • PIM Bi-Dir requirement on IPN
Day 1 Module 4
ACI Acceleration Series ACI Remote Leaf


CiscoPublic
Confidential
RL
ACI Remote Leaf Satellite DC
Any Routed Network

RL
Brownfield DC
VM
Pod 1
RL
(Main DC)
Edge Compute
RL
Co-Location
MTU, OSPF, and DHCP Relay <= 300 ms RTT, 100M+ BW On-premises APIC not required
Multicast is not required Up to 64 Remote Leaf Pairs Automated L2 VXLAN extension
Remote Leaf
Architecture Overview
Remote location contains Cisco Nexus® 9300 switches
connected to IP network and fully managed by APIC
cluster at the main data center
IP Network L2/L3
APIC and Spine Nodes remain at
main data center
Spine Remote Leaf Remote Leaf

APICs
vSwitch
Hypervisor
`
Bare
Leaf Metal
Remote Leaf Location (no

ACI Main Data Center spines needed)
Local traffic forwarding between endpoints
Remote Leaf use-cases
ü Satellite Data Center ü Mini POP locations at Co Lo Facility

ü Extension or Migration of DCs ü Telco 5G distributed DC
ü Disaster recovery *
IP Network
vSwitch Bare-
Hypervisor Metal
PBR
ACI Main DC Remote Location
*Data plane and control plane independency from main Pod
ACI 4.1.2+
Remote Leaf architecture evolution

Remote Leaf Location 2
Traffic between Remote Leaf switches is directly
forwarded vSwitch
Hypervisor
Bare
`Metal
IP Network
vSwitch
Hypervisor
Bare
Metal
ACI Main Data Center Remote Leaf Location 1

Remote Leaf works with Multi-Pod
IP Network
Remote Leaf Remote Leaf

Spine
Leaf
Remote Leaf Location – APICs
Remote Leaf Location –
Pod 1 Pod 2
Main Data Center – Main Data Center –

Pod1 Pod 2
Remote Leaf integration with ACI Multi-Pod is supported

Remote Leaf Scalability
Remote Leaf maximum switches allowed
ACI 4.2 128
ACI 3.2, 4.0, 4.1 40
ACI 3.1 30
Day 1 Module 5
ACI Acceleration Series Nexus Dashboard Orchestrator


CiscoPublic
Confidential
Automated Connectivity
Consistent Operations
Nexus Dashboard Orchestrator
Main functions handled by NDO:

REST
API
GUI • Provisioning of Day 0 connectivity between sites
• Control Plane (MP-BGP)
Nexus Dashboard
Orchestrator
• Data Plane (VxLAN)
• Central place to define and provision network and

security policies across sites
Hypervisor • Monitoring the health state of the different ACI sites
1 sec RTT
(max) • Can be automated with opensource tools!
…..
Site 1 Site 2 Site n
ACI Multi-Site
NDO Schemas and Templates
Schema
§ Template = ACI policy definition Lives only in Site 1 Lives in Site 1 and Site 2
Tenant1 Tenant1
(ANP, EPGs, BDs, VRFs, etc.) Stretched
Template
§ Schema = container of Templates sharing a
common use-case
• As an example, a schema can be dedicated to a
Tenant
§ The template is currently the atomic unit of
change for policies
• Such policies are concurrently pushed to one or
more sites
Site 1 Site 2
§ Scope of change: policies in different
EFFECTIVE
templates can be pushed to separate sites at EFFECTIVE
POLICY POLICY
different times
ACI Multi-Site
NDO Schema and Templates
§ Flexible way of referencing policy and objects

Schema 1 Schema 2 • Inside the same template
• Across templates in the same schema
Template 1 (Tenant 1) Template 1 (Tenant 2)
• Across templates in different schemas
EPG1 BD1 EPG2 BD2 § All objects defined inside a schema are visible and can be
referenced via the drop-down list
§ In order to properly plan how to deploy objects, be aware

Template 2 (Tenant common) Template 2 (Tenant 1)
of the following scalability values
• 10 templates per schema from MSO 2.2(4) release, 5 before
VRF1 C1 that
• 500 objects per schema supported up to MSO 2.2(2), 1000
objects per schema from MSO 2.2(3)
• Every object that can be defined in a template counts (EPGs,
BDs, VRFs, Contracts, etc.)
Nexus Dashboard Orchestrator
Green and Brown Field Flexibility
Green Field Deployment Import Policies from an Existing Fabric
Site 1
Site 1 Site 1 Site 1
Site 2
Site 1 Site 2 Site 1 Site 2

Green Field Green Field Existing Fabric Green Field
Day 1 Module 6
ACI Acceleration Series ACI Multi-Site


CiscoPublic
Confidential
ACI Multi-Site
Architecture VXLAN
MP-BGP - EVPN
Site 1 Site 2
…
Nexus
Dashboard
Orchestrator
GUI
Region 1 Region 2
Single Multi-Site Orchestrator domain
• Separate ACI Fabrics with independentAPIC clusters • Standard MP-BGP EVPN control plane between sites
• Nexus Dashboard Orchestrator is your central point for inter-site • Consistent Data plane VXLAN encapsulation across sites
connectivity and for designing / deploying policy across sites • L3 or L2 stretching across sites is possible
• Longer distances (<1s RTT max from NDO - APIC); up to 12 sites • No latency limitation between fabrics
ACI Multi-Site Use Cases
Scale-up model to build a large intra-DC
Data Center Interconnect (DCI)
network (above 400 leaf nodes)
Single Large Fabric

London Tokyo
ACI Multi-Site Los Angeles
Delhi
Site 1 Site 2 Site 3 Site N
Can also work with Multi-Pod for max sale
ACI Multi-Site
Software and Hardware Requirements
• ACI Multi-Site introduced from release 3.0(1)
• Support all ACI leaf switches (1st Generation, -EX and -FX) Can have only a subset of
Inter-Site Network spines connecting to the IP
• Only –EX spine (or newer) to connect to the ISN (ISN) network
• New 9364C/9332C non modular spine

1st Gen 1st Gen -EX -EX
(64/32 40G/100G ports) also supported
• 1st generation spines (including 9336PQ)
not supported
• Can still leverage those for intra-site leaf
to leaf communication
NDO
Multi-Site Network Options
ACI Multi-Site
Layer 3 networking scenario
• Layer-3-only connectivity across sites • Bridge Domains and subnets not extended across sites
• No Layer 2 extension or flooding is allowed
Site 1 Site 2
Nexus
Dashboard
Orchestrator
ACI Multi-Site
• Inter-VRF communication source and destination bridge domains: Different VRF instances
• Intra-VRF communication source EPG and destination EPG: Different bridge domains
Nexus Dashboard
Orchestrator
Site 1 Site 2
Tenant-A
VRF1 VRF2
BD-Red and
Subnet 1 Contract C1
EPG-Red BD-Green and
Subnet 2
EPG-Green
ACI Multi-Site
• Layer 2 connectivity across sites without flooding
• No Layer 2 BUM flooding across sites
• Same IP subnet defined in separate sites
Site 1 Site 2
Nexus Dashboard
Orchestrator
IP Mobility
ACI Multi-Site
• Layer 2 connectivity across sites without flooding (logical view)
• Objects to be stretched across the sites
• Layer 2 broadcast flooding is localized at each site
Nexus Dashboard
Orchestrator
APIC Site 1 APIC Site 2

No Layer 2 broadcast extension
Tenant A
VRF A
BD1 and Subnet 1
Web-EPG
Contract
BD2 and Subnet 2

App-EPG
ACI Multi-Site
• Layer 2 connectivity across sites with flooding
• Broadcast flooding is enabled across Fabrics
• Tenant and VRF are stretched between sites
Site 1 Site 2
Nexus Dashboard
Orchestrator
ACI Multi-Site
• Layer 2 connectivity across sites with flooding (logical view)
• BUM (Broadcast Unicast and Multicast) flooding is enabled across sites.
Nexus Dashboard
Orchestrator
APIC Site 1 Layer 2 broadcast extension

APIC Site 2
Tenant A
VRF A
BD1 and Subnet 1
Web-EPG
Contract
BD2 and Subnet 2

App-EPG
MP-BGP EVPN VXLAN
Multi-Site ACI Nexus Dashboard

Orchestrator Any Routed
On-Prem Network (ISN)
(with ISN)
Not configured ISN

by Nexus Dashboard
E1/1.4 E1/1.4 E1/1.4 E1/1.4
First Hop
OSPF (VLAN 4) OSPF (VLAN 4)
E1/1.4 E1/1.4 E1/1.4 E1/1.4

Spine
Managed & configured
by Nexus Dashboard Orchestrator Nodes
No need to connect all spines to ISN

ACI site 1 ACI site 2
Consider 50+ byte increase in ISN payload MTU (VXLAN) Tokyo Bangkok
MP-BGP EVPN VXLAN
Multi-Site ACI Nexus Dashboard

Orchestrator Any Routed
On-Prem Network (ISN)
(with ISN)
1-click encryption*
Not configured ISN

by Nexus Dashboard
E1/1.4 E1/1.4 E1/1.4 E1/1.4
First Hop
E1/1.4 E1/1.4 E1/1.4 E1/1.4

2 Anycast VTEP 2 Anycast VTEP Spine

Managed & configured Addresses Addresses
by Nexus Dashboard Orchestrator
1 for Unicast & 1 for Multicast (BUM
Traffic)
1 for Unicast & 1 for Multicast (BUM
Traffic)
Nodes
EVPN EVPN EVPN EVPN
Router ID Router ID Router ID Router ID
Multi-Pod ACI Sites supported. ISN/IPN can be shared

Control Plane MTU (BGP) can be adjusted on NDO Tokyo Bangkok
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential *Optional
Schema: Production Template: Stretched Network
Tenant C EPGs Contracts

Multi-Site ACI
On-Prem Logical Network Configuration
(Behind the scenes)
Nexus Dashboard
Orchestrator
EPG and VNID normalization (programmed at the spine layer)

MP-BGP EVPN VXLAN
EPG EPG PC Tag=1 PC Tag=2 EPG EPG
Routed Network
(Underlay)
Physical Network Configuration Physical Network Configuration
Shadow EPGs and contracts automatically created ACI site 1 ACI site 2
Tokyo Sri Lanka
Automate Cloud and DC Interconnect. Provision once, deploy anywhere
Schema: Production Template Stretched Network
Tenant C EPGs Contracts

Multi-Site ACI Template Tokyo
On-Prem Tenant A EPGs Contracts Logical Network Configuration
Nexus Dashboard
Logical Network Configuration Orchestrator
MP-BGP EVPN VXLAN

Import
Brownfield Logical Configuration Routed Network

(Underlay)
As sites are added, you may import their tenants
You may also import their objects in a template ACI site 1 ACI site 2
Tokyo Sri Lanka
This consolidates logical network management
Schema: Production Template Stretched Network
Tenant C EPGs Contracts Template Sri Lanka

Multi-Site ACI Template Tokyo
On-Prem Tenant A EPGs Contracts Logical Network Configuration Tenant B EPGs Contracts
Nexus Dashboard
Logical Network Configuration Logical Network Configuration
Orchestrator
MP-BGP EVPN VXLAN
Routed Network
(Underlay)

Tokyo Sri Lanka
CONFIGURATION
STEPS
Multi-Site ACI
Nexus Dashboard
Orchestrator
1 Run Nexus Dashboard Orchestrator
2 Setup ISN and add ACI Sites to NDO
3 Create Schema and Logical Network Configuration
88
Multi-Site ACI Business continuity/DR/DRaaS DC/Cloud Interconnect VM Mobility & Cloud Migration
On-Prem & Cloud
Nexus Dashboard
Can be run as
Physical Appliance Virtual Appliance SaaS*

3 physical nodes 3 VMs
150 ms RTT between 150 ms RTT between
nodes VMs
On-Prem/Cloud
Previously known as Multi-Site Orchestrator
Orchestrator can be enabled as a service in the Nexus Dashboard
Best practice: run NDO with the latest version
* Roadmap
Multi-Site ACI
Nexus Dashboard
Orchestrator
1 Run Nexus Dashboard Orchestrator
90
Nexus Dashboard
Orchestrator Multi-site ACI
Enable Multi-Site
Setup ISN (OSPF) & prepare
1 ACI Sites 2 Add ACI Sites to Orchestrator 3 Setup IP Addresses for BGP and
OSPF running on Spines
Multi-Site ACI
It is recommended to preserve APIC
Setup ISN (OSPF) Any Routed CoS/DSCP marking in the ISN
Network (ISN)
E1/49
ISN is Manually configured 172.16.121.0/30 ISN First Hop
E1/51.4 E1/51.4
172.16.111.2/30 172.16.222.2/30
MTU 9000 MTU 9000

Area 0 Area 0
Nexus Dashboard E1/61.4

172.16.111.1/30
E1/61.4
172.16.222.1/30
Spine Nodes
Orchestrator MTU 9000 MTU 9000

Miami San Jose
Consider 50+ byte increase in payload MTU (VXLAN)
VTEP Pools may be the same on each site
Nexus Dashboard
Orchestrator Multi-site ACI Setup ISN
devices
Configure sub-interface (VLAN 4) for interface facing Spine(s) and increase MTU (all links)
Setup ISN (OSPF) & prepare
1 ACI Sites
Enable OSPF on sub-interface and in the external-facing ISN links (or re-distribute)
It is recommended to assign a specific VRF for ISN traffic in your ISN device
It is recommended to match the QoS CoS mappings from the ACI fabric
Nexus Dashboard
1 Setup ISN (OSPF) 2 Add ACI Sites to NDO
Nexus Dashboard
Orchestrator Multi-site ACI Add Sites to Nexus
Dashboard Orchestrator
2 Add ACI Sites to NDO Provide the credentials for each site’s APIC in Nexus Dashboard
Add an ID to each site in Nexus Dashboard Orchestrator
Pin each ACI Site’s location running on-prem into the map
Nexus Dashboard
Enable Multi-Site
1 Setup ISN (OSPF) 2 Add ACI Sites to NDO 3 Setup IP Addresses for BGP and
Nexus Dashboard
Orchestrator Multi-site ACI Enable Multi-Site
Enable Multi-Site
3 Setup IP Addresses for BGP and
Enable Multi-Site on each Site
Configure anycast VTEP (unicast and multicast) & Router ID for MP-BGP and your Spine(s) OSPF L3 Out
Verify the automated configuration on each ACI Site
Multi-Site ACI MP-BGP EVPN VXLAN
Enable Multi-Site Any Routed
Network (ISN)
ISN First Hop

Area 0 Area 0
Nexus Dashboard E1/61.4

172.16.111.1/30
E1/61.4
172.16.222.1/30
Spine Nodes
Orchestrator MTU 9000 MTU 9000
Unicast VTEP: 172.1.1.10 Unicast VTEP: 172.2.1.10

Multicast VTEP (BUM): Multicast VTEP (BUM):
172.1.1.11 172.2.1.11
BGP EVPN Router ID BGP EVPN Router ID

172.1.1.12 172.2.1.12
BGP AS on each ACI Site must be different
VTEP Pools/GiPO may be the same on each site ACI site 1 ACI site 2
Miami San Jose
Automated configuration will show on tenant infra (intersite VRF)
Multi-Site ACI
Nexus Dashboard
Orchestrator
1 Run Nexus Dashboard NDO
99
Schema Design (Typical Deployment)
One Template per Site, plus a ‘Stretched’ Template
Schema ACME Site 1
ANP1 Site 1 Template

(Tenant1)
EPG1 EPG2 BD1 BD2
ANP1 Site 2 Template Site 2

(Tenant1)
EPG3 EPG4 BD3 BD4
ANP1 Site 3 Template

(Tenant1)
EPG5 EPG6 BD5 BD6 Site 3
ANP1 VRF
BD7 C1 C2
EPG7
Contracts
Stretched Template (Tenant1)

ACI Multi-Site and Border Leaf L3Outs
Deployment Options
Dedicated pair of WAN edge routers Shared pair of WAN edge routers
ISN ISN
WAN WAN
§ BLs on each ACI site connect to a separate pair of WAN edge routers § BLs of different sites connect to a common pair of WAN edge
for communication with the WAN routers for communication with the WAN
§ Most common deployment model for ACI fabrics geographically § Typical deployment model when Multi-Site is used for scaling up
dispersed the fabric in a single DC location
Problem Statement
Behavior before ACI Release 4.2(1)
Supported Design
✓ Not Supported Design
❌
Inter-Site Network Inter-Site Network
X
L3Out L3Out L3Out

WAN, Mainframes, WAN,WAN

Mainframes,
FW/SLB, etc… FW/SLB, etc…
Note: the same consideration applies to both Border Leaf L3Outs

ACI Multi-Site and L3Out ACI 4.2(1)
Release
Support of Intersite L3Out
• Starting with ACI Release 4.2(1) it is possible for endpoints

in a site to send traffic to resources (WAN, Mainframes,
Inter-Site Network FWs/SLBs, etc.) accessible via a remote L3Out connection
• External prefixes are exchanged across sites via MP-BGP
VPNV4/VPNv6 sessions between spines
MP-BGP VPNv4/VPNv6
• Traffic will be directly encapsulated to the TEP of the

remote BL nodes
• The BL nodes will get assigned an address part of an additional
L3Out
Site 1
(configurable) prefix that must be routable across the ISN
• This routable TEP pool can be configured on MSO or on APIC
WAN, Mainframes,
WAN
FW/SLB, etc…
• Same solution will also support transit routing across sites
(L3Out to L3Out)
ACI Multi-Site and Intersite L3Out ACI 4.2(1)
Release
Supported Scenarios
Inter-Site Network Inter-Site Network
L3Out L3Out L3Out

WAN, Mainframes,
WAN
FW/SLB, etc…
WAN
WAN, Mainframes, WAN, Mainframes,
FW/SLB, etc… FW/SLB, etc…
• Endpoint to remote L3Out communication (intra-VRF) • Inter-site transit routing (intra-VRF)

• Endpoint to remote L3Out communication (inter-VRF) • Inter-site transit routing (inter-VRF)
Summary
ACI Multi-Site Automate Data Center Interconnect

(On-Prem) Configure once, deploy anywhere
1 2
Automate Extend and secure
• MP-BGP EVPN, VXLAN • Extend Layer 2 and 3 across sites
• Back-to-Back and ISN topologies • Over any routed network
• 1-click encryption • Centralize policy definition
• Phase—out changes • Import brownfield ACI configurations
• Integrate Multi-Pod and Multi-Site
Day 1 Module 7
ACI Acceleration Series L4-L7 Services and Other

Deep Dive into ACI Anywhere Considerations

CiscoPublic
Confidential
Multi-Pod and Multi-Site ACI considerations
Firewall Asymmetric Routing
Packets leave through one path and return through Result: Firewall drops traffic due to
a different one lack-of-session state
Host-Route Advertisement (HRA)
Stretched
Subnet
Advertise specific prefixes outside the fabric Web Server: Result: Maintain symmetric forwarding
1.1.1.1
No HRA:
HRA: Announce
Announce Prefix
BD Prefix No HRA: Announce BD Prefix
1.1.1.1/32
1.1.1.0/24 externally 1.1.1.0/24 externally
ACI: Asymmetric routing and failure avoidance
Policy-Based Redirection
PBR: Define through a contract which traffic should Result: Avoid firewall or L4-L7 device bottlenecks
be forwarded to a specific MAC or IP working in L1/L2 or L3
PBR: Only send HTTP traffic to

firewall for inspection, route the
rest through the fabric
HTTP
ICMP
ACI: Asymmetric routing and failure avoidance
Active-Standby Independent Pairs Clustered*

(only in Multi-Pod)
Nexus Dashboard Nexus Dashboard Nexus Dashboard

Orchestrator Orchestrator Orchestrator
Resilient PBR Service Node in each Site
* Clustered FW connectivity is not supported in Multi-Site currently
Summary
L4-L7 Increase High-Availability

Multi-Pod/Multi-Site Maintain consistent security across sites
1 2
High-Availability Security & Flexibility
• Integrate SLBs and GSLBs to increase • Reduce L4-L7 bottlenecks (PBR)
site redundancy • Provision once, enforce and re-direct anywhere
• Minimize sub-optimal routing &
enable seamless failover (PBR/HRA)
Online Labs
dcloud
dcloud.cisco.com
Lab Access:
Cisco Nexus Dashboard Orchestrator for
ACI Lab v1
Scenarios
•Create New Users
•Create New Sites
•Day-0 Infrastructure Configuration
•ACI Multi-Site Use Cases
What’s next? Review the content and more
/CiscoDataCenterMadeEasy
Take an official Cisco ACI Training

DCACI
Implementing Cisco ACI
DCACIA
ACI Implementing Cisco ACI Advanced
Get Certified on ACI and CCNP DC*

300-620 DCACI Exam
Implementing Cisco ACI
600-660 DCACIA
Implementing Cisco ACI Advanced
* CCNP through ACI Specialization requires passing both 350-601 DCCOR +300-620 DCACI
Cloud Networking

ACI_Acceleration_Advanced_V1.0

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ACI_Acceleration_Advanced_V1.0

Uploaded by

Copyright:

Available Formats

ACI Acceleration Series

Part 2: Deep Dive into ACI Anywhere

Huyen Duong, TME Technical Leader - CNBU

ACI Beyond The Single Data Center

2 Introduction to ACI Anywhere

ACI Remote Leaf

ACI Acceleration Series ACI Recap

© 2022 Cisco and/or its affiliates. All rights reserved. Cisco

EPG 10 c EPG 20 c EPG 30

• Seamless Workload Mobility Integrated VXLAN Overlay

Nexus 9000 Series

OSPF EIGRP BGP

Associate End-Points Associate Prefixes

BD: Web_BD EPG EPG BD: DB_BD

Define once, use anywhere

ICMP HTTP 1.1.100.100 2.2.200.100

ACI Acceleration Series Introduction to ACI

© 2022 Cisco and/or its affiliates. All rights reserved. Cisco

Enterprise DC 5G Telco Edge

Public Cloud / IaaS Enterprise Edge

Private Cloud Colo / Bare Metal Cloud

Remote Location On Premises Public Cloud

Security Everywhere Analytics Everywhere Policy Everywhere

Remote-Leaf Cloud ACI

ACI Acceleration Series ACI Multi-Pod

© 2022 Cisco and/or its affiliates. All rights reserved. Cisco

Single Site ACI Multi-Site ACI

Separate management and Single management and logical network

You can run ACI anywhere your business takes you

Single Site ACI

Single Pod Multi-Room Data Centers

Separate management and

You can run ACI anywhere your business takes you

(up to 50 msec RTT)

* 10G only with QSA adapters on EX spines

(up to 50 msec RTT)

* 10G only with QSA adapters on EX spines

* 10G only with QSA adapters on EX spines

IPN First Hop

Underlay OSPF or eBGP (VLAN

IPN First Hop Not configured nor

Underlay OSPF or eBGP (VLAN

No need to connect all spines to IPN

IPN First Hop

Underlay OSPF or eBGP (VLAN

Unconfigured Pod 1 Pod 2

IPN First Hop Not configured nor

Underlay OSPF or eBGP (VLAN

No need to connect all spines to IPN

IPN First Hop

Underlay OSPF or eBGP (VLAN

Managed and configured

IPN First Hop

Underlay OSPF or eBGP (VLAN

Managed and configured

A TEP Pool is needed for each Pod

IPN First Hop

MP-BGP EVPN VXLAN

Anycast gateway (BD Subnets) distributed across pods Pod 1 Pod 2

Multi-Pod ACI Include the following blocks in RP:

PIM BI-Dir is needed in MultiPod to extend Bridge Domain GIPo

IPN First Hop

MP-BGP EVPN VXLAN

You can distribute APICs across pods