Tecdcn 2002

Next Generation Data Center
Infrastructure
Brenden Buresh – Principal Architect

Thomas Scheibe – VP Product Management
Azeem Suleman – Principal Technical Marketing Engineer
Matthias Wessendorf – Technical Marketing Engineer
TECDCN-2002
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Who We Are?
Azeem Suleman
Principal Engineer TME
@azeem_suleman
Thomas Scheibe Brenden Buresh

Product Management Principal Architect
@thomas0002 @BrendenBuresh
Matthias Wessendorf
Technical Marketing
Engineer
@matteq4er
Agenda
• Introduction
• VXLAN Fabrics Fundamentals (NX-OS)
• NX-OS Programmability
• DCNM
• Application Centric Infrastructure (ACI)
• ACI Programmability
• Network Assurance & Insights
• Conclusion
• 08:30 – 10:30 (2 hours)
• 10:30 – 10:45 (Break)
• 10:45 – 12:45 (2 hours)
Coffee and Lunch • 12:45 – 14:30 (Lunch)
Breaks • 14:30 – 16:30 (2 hours)
• 16:30 – 16:45 (Break)
• 16:45 – 18:45 (2 hours)
Thomas Scheibe
Product Management
Introduction
How to Build the Network for the Cloud?
Increasing Demand on Networks
• Zero Trust / Identity

• Application Refactoring – bare
metal, VM, container, function
• Hybrid/ Multi-Cloud
• Scalable Multi-tenancy models
• Automation
• Contextual Visibility
Network Perspective
Connectivity, Infrastructure
Network Perspective Application Perspective
Connectivity, Infrastructure Endpoints, Compute & Storage Tiers
Security Perspective
Zero Trust
Automation
Security Perspective
Zero Trust
Workload Context is Key
Application Policy Automation

Performance
Application Intent
Deployment Lifecycle
Application
Security
Analytics
Application Automation Infrastructure Automation
Abstraction Layer – Group Based Policy
Profile
EPG EPG EPG
Consumer/ Consumer/
Provider Contracts Provider Contracts
EPG (End Point Group) = Security Zone, App Tier, Physical Location, ..
Zero Trust
Segment based on Identity
1. End-end segmentation Compliance monitoring

needed Cloud 1
2. Flexibility to enforce in Cloud 2

Cloud 3
underlay and overlay
3. Mapping of domains/
tenants
Container
Zero Trust

needed Cloud 1

Cloud 3
tenants
Container
Zero Trust

needed Cloud 1

Cloud 3
tenants
Public & Private cloud Bare metal Virtual Container Fabrics Traditional network
Network Stack
Assurance
and Insights
Day 2
Management
and Automation ACI NX-OS DCNM Day 1
Network and Policy

Infrastructure
Assurance and Insights
Assure intent
“Ensure the business needs
are consistently maintained”
• Proactive Guarantee Reliability

• Service oriented “Solve problems before they
impact business”
• Correlate Infra and Flows
• Deep Visibility
2 Troubleshoot intelligently
“Highlight the needle in the
haystack”
Open Networking: Native API Integration
Auto Scale AWS API

EC2 and S3 Gateway
VMM Integration Cloud Native
Open Networking: Native API Integration
Auto Scale AWS API

EC2 and S3 Gateway
VMM Integration Cloud Native
“open = API use “open = anybody

has to be certified” can use”
Overlays, overlays, .. Cloud Scale
Dev to Ops: ACI CNI bridges the Gap
Developer Cloud(s) Data Center
1. Development in Cloud First

2. Deploy Anywhere
3. Generate composite ACI policy: app segmentation +
infrastructure security
Performance & Cost
Leverage Latest ASIC Technology
Server Silicon Cloud Scale Technology

X86 Processors 14nm 16nm ASICs ->7nm
Road to 400G+ 400G is deployed
Optics
100G QSFP 50/400G 100/400G

10/40G
QSFP DD QSFP DD
Switch
10/40G 25/100G 50/400G 100/400G
ASIC Technology
28nm 14/16nm 7nm
2017 2018 2019 2020
Pluggable Multispeed Interfaces
SFP QSFP
Pluggable Options Pluggable Options

• 100M SFP • 100M SFP (via QSFP)
• 1G SFP • 1G SFP (via QSA)
• 10G SFP+, Twinax, AOC • 10G SFP+, Twinax, AOC (via QSA)
• 25G SFP+, Twinax, AOC • 25G SFP+, Twinax, AOC (via SLIC)
• 40G QSFP, Twinax, AOC
• 50G Twinax, AOC (via SLIC)
• 100G QSFP, Twinax, AOC
Host: 1/10/25G Network: 10/40/50/100G
QSFP DD – Avoiding the Sins of the Past
QSFP DD
Lessons from the Past
1G → 10/25G or 40G → 100G transitions resulted in same high volume form factor being adopted. Why?
QSFP DD
SFP SFP28 QSFP DD
1G to 10/25G XEN
XFP
Journey PAK
X2
SFP SFP28 QSFP QSFP28 QSFP DD
1G to 10/25G XEN
40G to 100G to
XFP CFP4 400G CFP
Journey PAK
Journey
X2 CPAK
CFP2
SFP SFP28 QSFP QSFP28 QSFP DD
1G to 10/25G XEN
40G to 100G to
XFP CFP4 400G CFP
Journey PAK
Journey
X2 CPAK
CFP2
o System & network requirements do not change. Same port density per RU to maintain proven fabric designs
o Limited impact on system ecosystem – strong leverage
o Multi-speed switch port options – slower optics in higher speed ports
Network Stack API
Assurance & Insights Deep Visibility
Policy Federation – Cloud

Multi Site & Domain Extension
Fabric Level API’s

Network Semantics
DCNM APIC (DCNM, APIC)
Application Semantics
(APIC)
Fabric (NX-OS) Fabric (ACI)
Device Level API’s
Evolving Network Designs Routed Fabric
Traditional 3 Tier
VXLAN Bridging / Routing
DC Network Design
• VXLAN Flood & Learn
• VXLAN EVPN
• Separate Management Tools
(e.g. Nexus Fabric Manager)
DC Core
DC
PODs VXLAN Routing / Segmentation
APIC
• VXLAN Routing
• Policy Controller (APIC)
VPC in Access • Consistent policy across physical
Routed Aggregation & Core and virtual network
• Multi-hypervisor (VMware, MSFT,
OVS)
• Endpoint agnostic (bare metal, VM,
container)
Brenden Buresh
VXLAN Fabrics Principal Architect
Fundamentals
(NX-OS)
VXLAN Introduction
Data Center “Fabric” Journey (Standalone)
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2
Baremet al Hypervisor Hypervisor Hypervisor Baremet al Hypervisor Baremet al Baremet al Hypervisor Hypervisor
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Overlay Based Data Center: Edge Devices
Network Overlays Host Overlays
Overlay Overlay
VTEP VTEP VTEP VTEP - - - -
VTEP VTEP VTEP VTEP

Baremetal Baremetal Baremetal Baremetal Hypervisor Hypervisor Hypervisor Hypervisor
• Router/Switch End-Points
• Virtual End-Points only
• Protocols for Resiliency/Loops
• Single Admin Domain
• Traditional VPNs
• VXLAN, NVGRE, STT
• VXLAN, OTV, VPLS, LISP, FP
Overlay Based Data Center: Edge Devices
Network Overlays Host Overlays
Overlay Overlay
VTEP VTEP VTEP VTEP - - - -
Hybrid Overlays
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal Baremetal Hypervisor Hypervisor Hypervisor Hypervisor
• Router/Switch End-Points
Overlay • Virtual End-Points only
• Protocols for Resiliency/Loops
• Single Admin Domain
• Traditional VPNs
• VXLAN, NVGRE, STT
•
- - VTEP VTEP
VXLAN, OTV, VPLS, LISP, FP
VTEP VTEP
Hypervisor Hypervisor Baremetal Baremetal
• Physical and Virtual

• Resiliency and Scale
• Cross-Organizations/Federation
• Open Standards
Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3
Data-Plane
Control-Plane
• Overlay Layer-2/Layer-3 Unicast Traffic
• Peer-Discovery
• Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution
• Local Learning
Multicast traffic (BUM traffic) forwarding
• Ingress Replication (Unicast)
• Remote Learning
• Multicast
Overlay Taxonomy - Underlay
Spine Spine Spine Spine
Underlay
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Underlay
LAN
Segment
Underlay
LAN
Segment
Virtual
Server Physical
Server
Layer-3
Interface Spine Spine Spine Spine
Peering
Underlay
LAN
Segment
Virtual
Server Physical
Server
Overlay Taxonomy - Overlay
Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
LAN
Segment
Virtual
Server Physical
Server VTEP: VXLAN Tunnel End-Point
VNI/VNID: VXLAN Network Identifier
Overlay Taxonomy - Overlay
Tunnel Encapsulation
Spine
(VNI Namespace)
Spine Spine Spine
Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
LAN
Segment
Virtual
Server Physical
Server VTEP: VXLAN Tunnel End-Point
VNI/VNID: VXLAN Network Identifier
Introducing VXLAN
MAC 802.1q IP Payload CRC

Src and Dst
Src, Dst VTEP VTEP IP
and Hop-by- UDP Dst VXLAN
Address Port 4789 VNI
Hop MAC Original Layer-2 Frame
Data-Plane
Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
(VXLAN)
14-byte + 20-byte +8-byte + 8-byte* = 50 Bytes
of total overhead UDP Src Port
Hash of L2/L3/L4
headers of
original Frame
*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header
VXLAN Frame Format – MAC in IP Encapsulation
Field Value Bites Total Field Value Bites Total

Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16
(4 Bytes Optional)
Src. MAC Address Next-Hop MAC Address 48
8 Bytes
Destination Port 4789 (UDP) 16
14 Bytes
VLAN Type 0x8100 16 UDP Length 16
VLAN ID Tag 16 Checksum 0x0000 16
Ether Type 0x0800 16
Outer
Outer IP UDP VXLAN Inner MAC Payload CRC
MAC
Field Value Bites Total

Field Value Bites Total
IP Header Misc. Data 72
VXLAN Flags RRRRIRRR 8
Protocol 0x11 (UDP) 8
20 Bytes
8 Bytes
Reserved 24
Header Checksum Various 16
VNI 16M Possible Segments 24
Source IP Src, VTEP IP 32
Reserved 8
Destination IP Dest. VTEP IP 32
No Path Diversity
Spine
• Equal Cost Multi-Pathing (ECMP)
uses Header information to form
Path Diversity
Leaf
• Some Tunnel Protocol provide no Leaf
Baremetal
AS#65500 diversity in IP or Protocol Header Baremetal
• As a Result, all Packets travel

the same Path
Spine
• No Path Diversity or Entropy
Introducing VXLAN – Entropy
Spine
• VXLAN provides variable UDP
Source Port in Outer Header
• Hash of the inner Layer-2/Layer-
VTEP VTEP
Baremetal
AS#65500 3/Layer-4 Headers of the original Baremetal
Ethernet Frame.
• Enables entropy for ECMP Load
Spine
balancing in the Network
Introducing VXLAN – Entropy
Spine
VTEP VTEP
AS#65500
Entropy Spine
happens here
Data-Plane (VXLAN)
A Scale Out Architecture
More Spine – More Bandwidth – More Resiliency

• Leaf Spine Spine Spine Spine
• Smallest Operational Entity
• Spines
• Wide vs. Big
• Uplinks
• Symmetric to all Spines or Pods
More Leaf – More Ports – More Capacity
• SAYG: Scale as You Grow
Folded Clos Topology – Device Roles
• Spine
• Interconnecting Leafs and Border
Leafs Spine Spine Spine Spine
• Leaf (VTEP)
• Virtual Machines
• Physical Machines
•
FEXes
• 3rd Party Switches
• UCS FIs
• Blade Switches WAN
• Border Leaf (VTEP)

• External Connectivity
• Spine
• Leaf (VTEP)
•
FEXes
• UCS FIs

• Spine
• Leaf (VTEP)
•
FEXes
• UCS FIs

• Spine
• Leaf (VTEP)
•
FEXes
• UCS FIs

• Border Spine WAN

• Leaf (VTEP)
• Physical Machines Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• FEXes
• UCS FIs
• Blade Switches

• Leaf (VTEP)
• FEXes
• UCS FIs
• Blade Switches

• Leaf (VTEP)
• FEXes
• UCS FIs
• Blade Switches
The Super-Spine
POD 1
The Super-Spine
Spine Spine Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
POD 1 POD 2
The Super-Spine
SuperSpine
SuperSpine SuperSpine
POD 1 POD 2
The Super-Spine
SuperSpine
• Scale Out
SuperSpine SuperSpine
• Not Limited to Port Density

• Simpler Capacity Planning
• Beyond a Single Server Room

• Allows Interconnecting Pods
• Retains Intra-Pod Topology with

Flexible Inter-Pod Connectivity
POD 1 POD 2
VXLAN Gateway Types
Egress packet is IEEE 802.1q
Ingress VXLAN packet tagged interface. packet is
on RED segment VTEP BRIDGED to new VLAN
• VXLAN to VLAN Bridging

• (Layer-2 Gateway) VXLAN Layer-2
Gateway
VXLAN Gateway Types

Gateway

on RED segment VTEP ROUTED to new VLAN
• VXLAN-to-VLAN Routing
• (Layer-3 Gateway)
VXLAN Router
VXLAN Gateway Types

Gateway

on RED segment VTEP ROUTED to new VLAN
• VXLAN-to-VLAN Routing
VXLAN Router
Ingress VXLAN packet Egress VXLAN packet is

on RED segment VTEP ROUTED to new segment
• VXLAN-to-VXLAN Routing
VXLAN Router
EVPN Deep Dive
What is VXLAN and EVPN ?
• VXLAN
• Standards based Encapsulation
• RFC 7348
• Uses UDP-Encapsulation
• Transport Independent
• Layer-3 Transport (Underlay)
• Flexible Namespace
• 24-bit field (VNID) provides ~16M
unique identifier
• Allows Segmentation
What is VXLAN and EVPN ?
• EVPN
• VXLAN
• Standards based Control-Plane
• Standards based Encapsulation
• RFC 7432
• RFC 7348
• Uses Multiprotocol BGP
• Uses UDP-Encapsulation
• Uses Various Data-Planes
• Transport Independent
• VXLAN (EVPN-Overlay), MPLS,
• Layer-3 Transport (Underlay)
Provider Backbone (PBB)
• Flexible Namespace
• Many Use-Cases Covered
• 24-bit field (VNID) provides ~16M
• Bridging, MAC Mobility, First-Hop
unique identifier
& Prefix Routing, Multi-Tenancy
• Allows Segmentation
(VPN)
Introducing Ethernet VPN (EVPN)
EVPN MP-BGP – RFC 7432
MPLS Provider Backbone Bridges Overlay (NVO3)
(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)
Introducing Ethernet VPN (EVPN)
EVPN MP-BGP – RFC 7432
MPLS Provider Backbone Bridges Overlay (NVO3)
(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)
VXLAN and EVPN Related RFCs & Drafts (IETF)
ID Title Category
RFC 7348 Virtual Extensible Local Area Network Data Plane
RFC 7432 BGP MPLS based Ethernet VPNs Control Plane
draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane
draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane
draft-ietf-bess-l2vpn-evpn-prefix-
IP Prefix Advertisement in E-VPN Control Plane
advertisement
draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane
EVPN Layer-2 Services (1)
Single Subnet per EVI Multiple Subnets per EVI

• VLAN-based • VLAN-aware
• Per EVI BGP Route Distinguisher / Router Target per EVI / VNI
• BGP Route-Target constrain mechanism to limit propagation (import/export)
• 1:1:1 mapping • 1:1:N mapping

• VNI to EVI to Single Broadcast • VNI to EVI to Multiple Broadcast
Domain (Bridge Domain) Domains (Bridge Domains)
• Ethernet Tag ID must be 0 • Ethernet Tag ID is to differentiate
Bridge Domains
VID = VLAN ID
VNI = VXLAN Network Identifier
EVI = EVPN Virtual Instance TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
EVPN Layer-2 Services (1a)
(draft-ietf-bess-evpn-overlay – Section 5.1.2)
• VLAN-based
VID
10 VNI EVI
Route Target: 65000:30000
VID = VLAN ID
EVI = EVPN Virtual Instance © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN Layer-2 Services (1a)
(draft-ietf-bess-evpn-overlay – Section 5.1.2)
• VLAN-based
VID
10 VNI EVI
VID = VLAN ID
EVPN Layer-2 Services (1b)
(draft-ietf-bess-evpn-overlay – Section 6.1)
• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VID
10 VNI EVI
VID = VLAN ID
EVPN Layer-2 Services (1b)
(draft-ietf-bess-evpn-overlay – Section 6.1)
• VLAN-based
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VID
10 VNI EVI
VID = VLAN ID
EVPN Layer-2 Services (1c)
(RFC 7432 – Section 6.3)
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
VID
10
VID
VNI EVI VNI EVI
VID
10 20
VID
30
[2]:[0]:[20]:[48]:[0050.569f.d495]:[32]:[192.168.22.33]
EVPN IP-VRF Services (2)
Interface-Less Model Interface-Full Model (2 Modes)

• Route-Type 5 only • Core-facing IRB
• Next-Hop is remote VTEP • Unnumbered Core-facing IRB
(Optional)
• Two extended communities
• Encapsulation Extended Community • Route-Type 5
• Router’s MAC Address (remote VTEP) • Next-Hop is remote IRB
• One or two extended communities
• Encapsulation Extended Community
• Router’s MAC Address (remote VTEP)
• Route-Type 2
• Containing Router MAC or MAC/IP
Route Type 2 = MAC/IP Route

Route Type 5 = IP Prefix Route
EVPN IP-VRF Services (2a)
(draft-ietf-bess-evpn-prefix-advertisement – Section 5.4.1)
• Interface-Less
NVE IP: 10.22.22.34
EVPN Router MAC: 0200.0ADE.DE22
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IP-VRF Services (2a)
• Interface-Less
NVE IP: 10.22.22.34
VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
EVPN IP-VRF Services (2b)
• Interface-Less • Interface-Full (Core-facing IRB)
NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

VTEP VTEP VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop) BGP 10.22.22.34 (Next-Hop)
Update Encap:8 (VXLAN) Update Encap:8 (VXLAN)
Router MAC:0200.0ade.de22 Router MAC:0200.0ade.de22
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
EVPN IP-VRF Services (2b)
• Interface-Less • Interface-Full (Core-facing IRB)
NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

VTEP VTEP VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
EVPN IP-VRF Services (2c)
• Interface-Less • Interface-Full
(Unnumbered Core-facing IRB)
NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

VTEP VTEP VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
EVPN IP-VRF Services (2c)
• Interface-Less • Interface-Full
(Unnumbered Core-facing IRB)
NVE IP: 10.22.22.34 NVE IP: 10.22.22.34

VTEP VTEP VTEP VTEP
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0] [5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
EVPN IRB Services (3)
Symmetric Inter-Subnet Asymmetric Inter-Subnet

Forwarding Forwarding
• Bridge->Route/Route->Bridge • Bridge->Route->Bridge
• Symmetric VNI in both • Different (Asymmetric) VNI
directions depending on directions
• Adjacency contains Remote • Adjacency contains Remote
VTEP,VRF VTEP,VRF and End-Points
• Optimal for Scale • Potential Sub-Optimal for Scale
• Flexible Configuration • Consistent Configuration
VTEP = VXLAN Tunnel End-Point

VRF = Virtual Routing and Forwarding
VNI = VXLAN Network Identifier 53
TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Services (3a)
(Traditional Bridging – Depending on EVPN Layer-2 Services)
• Symmetric IRB • Asymmetric IRB

VNI 30000 (L2VNI) VNI 30000 (L2VNI)
V1 V2 V1 V2
MA MA
MAC IP IP MAC MAC IP IP MAC
C C
192.168.22.33 192.168.22.44 192.168.22.33 192.168.33.44

192.168.22.44
Bridge Bridge
EVPN IRB Services (3b)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4 and 5)

V1 V2 V1 V2
MA MA MA MA
IP IP IP IP
C C C C
192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44
Bridge -> Route -> Route -> Bridge Bridge -> Route -> Bridge
EVPN IRB Services (3b)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4 and 5)

VNI 30000 (L2VNI)

V1 V2 V1 V2
MA MA MA MA
IP IP IP IP
C C C C
192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44
Bridge -> Route -> Route -> Bridge Bridge -> Route -> Bridge
EVPN IRB Services (3c)
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MA MA
IP IP
C C
192.168.22.33 192.168.33.44
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
EVPN IRB Services (3c)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MA MA
IP IP
C C
192.168.22.33 192.168.33.44
[5]:[0]:[0]:[24]:[192.168.22.0]:[0.0.0.0]
BGP 10.22.22.34 (Next-Hop)
EVPN IRB Services (3d)

VNI 30000 (L2VNI)

V1 V2 V1 V2
MA MA MA MA
IP IP IP IP
C C C C
192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

[5]:[0]:[0]:[24]:[192.168.22.0]:[10.22.22.34]
BGP
10.22.22.34 (Next-Hop)
Update
Encap:8 (VXLAN)
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
EVPN IRB Services (3d)

VNI 30000 (L2VNI)

V1 V2 V1 V2
MA MA MA MA
IP IP IP IP
C C C C
192.168.22.33 192.168.33.44 192.168.22.33 192.168.33.44

[5]:[0]:[0]:[24]:[192.168.22.0]:[10.22.22.34]
BGP
10.22.22.34 (Next-Hop)
Update
Encap:8 (VXLAN)
[2]:[0]:[0]:[48]:[0200.0ade.de22]:[32]:[10.22.22.34]
BGP 10.22.22.34 (Next-Hop)
EVPN IRB Services (3e)
• Symmetric IRB and Asymmetric IRB is NOT interoperable per-se

• Routing is implemented different
• Symmetric IRB: Bridge -> Route -> Route -> Bridge
• Asymmetric IRB: Bridge -> Route -> Bridge
• Symmetric IRB uses Route-Type 2 with two VNI
• L3VNI for routing and L2VNI for bridging
• Asymmetric IRB uses Route-Type2 and Route-Type 5
• Type 2 with L2VNI for bridging and inter-subnet forwarding (known VNI/VTEP)
• Type 5 with L3VNI for inter-subnet forwarding (see IP-VRF Services)
• If implemented
Conclusions - Cisco’s EVPN Implementation
• VLAN to VNI to EVI • Multiple VLAN to VNI to EVI
• More granular control (route-target) • No true MAC separation (potential)
• Interface-Full
• Interface-Less
• Additional overhead (2 routes and additional
• Follows classic routing lookup)
• No need for MAC routes in routing
• Asymmetric IRB
• Symmetric IRB • Extensive usage of Adjacency Tables
• More Centralized Gateway-like
• Adjacency Tables are preserved
• “Consistent” configuration necessary if Distributed
• Configuration is flexible Gateway is required
Conclusions - Cisco’s EVPN Implementation
• VLAN to VNI to EVI • Multiple VLAN to VNI to EVI
• More granular control (route-target) • No true MAC separation (potential)
• Interface-Full
• Interface-Less
• Additional overhead (2 routes and additional
• Follows classic routing lookup)
• No need for MAC routes in routing
• Asymmetric IRB
• Symmetric IRB • Extensive usage of Adjacency Tables
• More Centralized Gateway-like
• Adjacency Tables are preserved
• “Consistent” configuration necessary if Distributed
• Configuration is flexible Gateway is required
EVPN Operations
EVPN - Host and Subnet Route Distribution
• Host Route Distribution

RR RR decoupled from the Underlay
protocol
Overlay • Use MultiProtocol-BGP (MP-

BGP) on the Leaf nodes to
Leaf Leaf Leaf Leaf Leaf Leaf Leaf distribute internal Host/Subnet
Routes and external reachability
information
• Route-Reflectors (RR) deployed
for scaling purposes
EVPN Control Plane - Host and Subnet Routes
• BGP EVPN NLRI*

• Host MAC (Route Type 2)
• MAC only, Single VNI, Single
Route Target
Overlay
• Host MAC+IP (Route Type 2)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • MAC and IP, Two VNI, Two Route
Target, Router MAC
• Internal and External Subnet

Prefixes (Route Type 5)
• IP Subnet Prefix, Single VNI,
Single Route Target
*NLRI: Network Layer Reachability Information (BGP Update Format)
Host Advertisements
Overlay
Baremetal Baremetal Baremetal
Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
*L2VNI: VNI for all Bridging operation (”VLAN-VNI”) TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

Overlay

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104
Overlay

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
Host Advertisements
Type
Type MAC // Length
MAC Length L2VNI // RT
L2VNI RT IP / Length L3VNI / RT Next-Hop Seq.
22 0000.3001.1101 // 48
0000.3001.1101 48 3001, 65500:3001
3001, 65500:3001 10.200.200.101
22 0000.3001.1102 // 48
0000.3001.1102 48 3001, 65500:3001
3001, 65500:3001 10.200.200.104
Overlay • Host MAC (Route Type 2)

2 0000.3002.2101 / 48 3002, 65500:3002 10.200.200.107
• MAC
• MPLS Label1 (L2VNI*)
•
Route Target for MAC-VRF
• MAC attributes are Mandatory

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L2VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label1)
10.200.200.101 (metric 3) fromL2VNI
10.10.10.201 (10.10.10.201)
Encap:8
Route Target
Origin IGP, MED not set, localpref 100, weight 0
VXLAN
Received label 3001
Extcommunity: RT:65500:3001 ENCAP:8
Originator: 10.10.10.101 Cluster list: 10.10.10.201
Host Advertisements
Overlay

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
*L3VNI: VNI for all Routing operation (”VRF-VNI”)

Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101

Overlay

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104
Overlay

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

Host Advertisements
2 0000.3001.1101 / 48
Spine Spine
3001, 65500:3001
Spine Spine
192.168.10.101 /32 • Host MAC+IP
10.200.200.101(Route Type 2)
5000, 65500:5000
• MAC and IP
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104
• MPLS Label1 (L2VNI)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 192.168.20.101 /32 • Route
5000, 65500:5000 Target for MAC-VRF
10.200.200.107
• MPLS Label2 (L3VNI*)

Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Route Target for IP-VRF
• Router MAC
• IP Attributes are Optional

• Populated through ARP/ND
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
MAC/IP for VRF default, address family L2VPN EVPN
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,
version 4
Paths: (1 available, best #1) IP Address
Length
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked IP Address
Next-Hop L3VNI
IP Address
Path type: internal,L2VNI
path is (MPLS
valid, is best path, no labeled nexthop
Label2)
AS-Path: NONE, path
(MPLSsourced
Label1) internal to AS
10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 VXLAN
Received label 3001 5000
Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
L2VNI L3VNI
Route Target Router MAC
Route Target
Subnet Route Advertisements
Overlay
Subnet A
192.168.10.0/24
Type IP / Length L3VNI / RT Next-Hop Seq.
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

• Internal and External Subnet
Prefixes (Route Type 5)
Overlay • IP Prefix
• MPLS Label (L3VNI)
• Route Target for IP-VRF
• Router MAC
• Populated through External

Routing Protocol
Subnet A
192.168.10.0/24
Type IP / Length L3VNI / RT Next-Hop Seq.
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.104
Overlay
Subnet A Subnet A
192.168.10.0/24 192.168.10.0/24
Type IP // Length
IP Length L3VNI // RT
L3VNI RT Next-Hop Seq.
55 192.168.10.0 /24
192.168.10.0
Spine
/24
Spine
5000, 65500:5000
5000, 65500:5000
Spine Spine
10.200.200.101 • IP Prefix Learning
55 192.168.10.0 /24
192.168.10.0 /24 5000, 65500:5000
5000, 65500:5000 10.200.200.104
• via BGP with VRF-Lite
• via LISP on Nexus 7000/7700
5 192.168.20.0 /24
Overlay 5000, 65500:5000 10.200.200.107 • via other routing protocol (static
or dynamic)
• Default: Export of IP Host and IP

Prefix Routes advertisements
• Filter and Summarize where
appropriate
Subnet A Subnet A Subnet B
192.168.10.0/24 192.168.10.0/24 192.168.20.0/24
Ethernet Tag
Ethernet Identifier
Segment
V2# show bgp l2vpn evpn 192.168.10.0 (Ethtag)
Identifier (ESI) IP Address
Route Type: IP Address
Length family
IP Prefix for VRF default, address L2VPN EVPN
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Next-Hop
IP Address
Path type: internal, path
L3VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label)
10.200.200.101 (metric 3) fromL3VNI
10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 Router MAC
Route Target VXLAN
Received label 5000
Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)

3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Data-Plane (VXLAN)

3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Dst VTEP IP L2VNI Dst MAC Dst IP

10.200.200.101 3001 0000.3001.1101 192.168.10.101
Data-Plane (VXLAN)
Bridging

3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Dst VTEP IP L3VNI Router MAC Dst IP

10.200.200.101 5000 0200.0ade.de01 192.168.10.101
Data-Plane (VXLAN)
Routing
Routing and the Router MAC – Ethernet
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1
interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
SMAC DMAC SIP DIP

Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP

Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Router MAC
SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101 0000.3001.1101 0000.3002.2101 192.168.10.101 192.168.20.101
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Routing and the Router MAC – VXLAN
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1
interface: NVE1 interface: NVE1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
SMAC DMAC SIP DIP

Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP

Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Router MAC

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 0000:3002.2101 192.168.10.101 192.168.20.101 0000.3001.1101 0000.3002.2101 192.168.10.101 192.168.20.101
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Packet Walk – ARP Request
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
ARP Request for 192.168.10.102 Spine
SMAC: DMAC:
0000.3001.1101 FFFF.FFFF.FFFF
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
SMAC: DMAC: SIP DIP VXLAN SMAC DMAC ARP Request for
0000.3001.1101 FFFF.FFFF.FFFF 192.168.10.102
10.200.200.101 239.0.0.1 30001 0000.3001.1101 FFFF.FFFF.FFFF
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
SMAC: DMAC: SIP DIP VXLAN SMAC DMAC ARP Request for
0000.3001.1101 FFFF.FFFF.FFFF 192.168.10.102
10.200.200.101 239.0.0.1 30001 0000.3001.1101 FFFF.FFFF.FFFF
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
ARP Request for 192.168.10.102
SMAC: DMAC:
Host C
0000.3001.1101 FFFF.FFFF.FFFF MAC: 0000.3002.2101
IP: 192.168.20.101
Packet Walk – ARP Response
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
ARP Response for 192.168.10.102
SMAC: DMAC:
Host C
0000.3001.1102 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
SIP DIP VXLAN SMAC DMAC ARP Response for

192.168.10.102
10.200.200.103 10.200.200.101 30001 0000.3001.1102 0000.3001.1101
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
SMAC: DMAC:
Host C
0000.3001.1102 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
ARP Response for 192.168.10.102 Spine
SMAC: DMAC:
0000.3001.1102 0000.3001.1101
SIP DIP VXLAN SMAC DMAC ARP Response for
192.168.10.102
10.200.200.103 10.200.200.101 30001 0000.3001.1102 0000.3001.1101
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
SMAC: DMAC:
Host C
0000.3001.1102 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.20.101
Packet Walk – Bridging
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

Spine
Payload
10.200.200.101 10.200.200.103 30001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

Spine
Payload
10.200.200.101 10.200.200.103 30001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
SMAC DMAC SIP DIP Host C

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102 MAC: 0000.3002.2101
IP: 192.168.20.101
Packet Walk – Routing
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
SMAC DMAC SIP DIP Spine
0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

Payload
Spine
10.200.200.101 10.200.200.104 50001 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

Payload
Spine
10.200.200.101 10.200.200.104 50001 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
TOR3 TOR4
SMAC DMAC SIP DIP
VLAN 101 (Green)
Leaf Leaf
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Packet Walk – Routing (Silent Host)
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

10.200.200.101
TOR1
10.200.200.102 50001 0200.0ade.de01 0200.0ade.de07
TOR2
192.168.10.101 192.168.20.101
Payload
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

SIP DIP VXLAN SMAC DMAC ARP Request for
192.168.20.101
10.200.200.102 239.0.0.1 30002 AGM FFFF.FFFF.FFFF
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

SIP DIP VXLAN SMAC DMAC ARP Request for
192.168.20.101
10.200.200.102 239.0.0.1 30002 AGM FFFF.FFFF.FFFF
Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
ARP Request for 192.168.20.101
Host SMAC: DMAC:
AGM FFFF.FFFF.FFFF
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
VLAN 202 (Blue)

Spine
TOR3 TOR4
VLAN 101 (Green)
Leaf Leaf
Host
Host B
MAC: 0000.3001.1102
VXLAN EVPN
VLAN 202 (Blue)

IP: 192.168.10.102
Host
Host C
ARP Response for 192.168.20.101MAC: 0000.3002.2101
IP: 192.168.20.101
SMAC: DMAC:
0000.3002.2102 AGM
VXLAN Design
Considerations
Underlay Design
Review
MTU and Overlays
• Data Center often require Jumbo MTU

• Most Server NIC support up to 9000 Bytes
• Network Switches support MTU up to

9216* Bytes
• Accommodates Jumbo MTU plus Overlay
overhead
• Avoid Fragmentation
• Adjust the Transport Network with
appropriate MTU
*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic
MTU and Overlays
• Data Center often require Jumbo MTU

• Most Server NIC support up to 9000 Bytes
• Network Switches support MTU up to

9216* Bytes
• Accommodates Jumbo MTU plus Overlay
overhead
• Avoid Fragmentation
• Adjust the Transport Network with
appropriate MTU
*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic
Interface Principles
• Routed Ports and Interfaces Spine Spine Spine Spine
• Layer-3 Interfaces between Leaf

and Spine(no switchport)
• For each Point-2-Point (P2P)
connection, minimum /31 required
• Alternative, use IP Unnumbered Leaf Leaf Leaf Leaf Leaf Leaf Leaf
(/32)
• Use Loopback as Source-

Interface for VTEP (NVE*)
IP Addressing Principles
Routing
Identifier Rendezvous
Point
• Prepare a IP Addressing Plan p2p* Links / IP
Unnumbered
• Separate Interface functions Spine Spine Spine Spine
through IP Addressing
(Aggregates)
• Unicast Routing – Routing Protocol
Peering (p2p*) Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Unicast Routing – Routing Identifier

(RID)
• VTEP and VPC VTEP
Routing
• Multicast Rendezvous-Point (RP) Loopback Identifier
• IPv4 only (today) p2p Agg: 10.1.1.0/24

RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
Unicast Routing – OSPF and IS-IS
• OSPF – watch your Network Type

• IS-IS – what was this CLNS?
• Network Type Point-2-Point • Independent of IP (CLNS)
• Preferred (only LSA type-1) • Well suited for routed
• No DR/BDR election interfaces/ports
• Well suited for routed • No SPF calculation on Link change;
interfaces/ports (optimal from a only if Topology changes
LSA DB perspective) • Fast Re-convergence
• Full SPF calculation on Link • Not everyone is familiar with it
Change
Unicast Routing – BGP
• eBGP Underlay Routing –

Service Provider style
• Two Different Models
• Two-AS
• Multi-AS
• BGP is a Distance Vector

Protocol
• actually Path Vector Protocol
• AS* are used to calculate the
Path (AS_Path)
Unicast Routing – eBGP Two-AS Model
All-Spine AS#65500
• eBGP Two-AS, yes it works!
• eBGP peering for Underlay
• Spine is not a Route-Reflector
(eBGP) – Retain Route-Targets
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Disable BGP AS-Path check
• Next-Hop needs to be Unchanged
All-Leaf AS#65501 • Underlay is Reachability!

• Advertise your Loopbacks
Unicast Routing – eBGP Multi-AS Model
• eBGP Two-AS, yes it works!


• Disable BGP AS-Path check
• Underlay is Reachability!
• Changes Overlay Routing Policy

• Manually define Route-Targets
Unicast Routing – eBGP Multi-AS Model
All-Spine AS#65500 • eBGP Two-AS, yes it works!


• Disable BGP AS-Path check
• Underlay is Reachability!
• Changes Overlay Routing Policy

• Manually define Route-Targets
Unicast Routing – eBGP Model
Spine Spine Spine Spine • Two different BGP Peering

• Global IPv4/v6 Address-Family
• Use Physical Interface IP
• eBGP peering for Overlay

• Global EVPN Address-Family
• Use Loopback Interface IP
• BFD not so ok
Unicast Routing – eBGP Model
Spine Spine Spine Spine • Two different BGP Peering

• Global IPv4/v6 Address-Family
• Use Physical Interface IP
• eBGP peering for Overlay

• Global EVPN Address-Family
• Use Loopback Interface IP
• BFD not so ok
Multicast Enabled Underlay – PIM ASM
• PIM Any-Source-Multicast (ASM) RP RP

• Platform Support
• Nexus 9000 / Nexus 7000 (F3/M3)
• ASR 1000 / ASR 9000
Underlay
• RP Redundancy Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• PIM Anycast-RP or MSDP
• Source-Trees (Unidirectional)
• 1 Source Tree per VTEP per
Multicast Group
Underlay – PIM ASM with PIM Anycast-RP
RP RP
S,G
Underlay
RP RP
S,G Underlay
RP RP
S,G
Underlay
RP RP
Underlay
S,G
RP RP
Underlay S,G
Why Do I Need Multicast Again?
Destination Group
239.1.1.1
Spine (0100.5E01.0101)
Spine Spine Spine
3
MAC VNI VTEP MAC VNI VTEP
0000.3001.1101 30001 E1/12
Overlay
3 0000.3002.2101 30002 E1/4
2
SMAC: MAC_LEAF1
VTEP VTEP VTEP VTEP
ARP Request for 192.168.10.102 VTEP VTEP VTEP
DMAC: 0100.5E01.0101
Underlay
Src MAC: 0000.3001.1101
Dst MAC: FFFF.FFFF.FFFF SIP: IP_LEAF1
DIP: 239.1.1.1 ARP Request for 192.168.10.102
1 4 Src MAC: 0000.3001.1101
UDP Dst MAC: FFFF.FFFF.FFFF
VXLAN VNID: 30001

MAC VNI
VNI VTEP
VTEP
Overlay
ARP Request
Baremetal Baremetal 0000.3001.1102
0000.3001.110 30001
30001 E1/8
E1/8 Baremetal
SMAC: 0000.3001.1101 2
Host A DMAC: FFFF.FFFF.FFFF Host B 0000.3001.1101 30001 LEAF1 Host C
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Things to Remember
Multicast Enabled Underlay
• Multi-Destination Traffic (Broadcast, Unknown Unicast, etc.) needs to be

replicated to ALL VTEPs serving a given VNI
• Each VTEP is Multicast Source & Receiver
• For a given VNI, all VTEPs act as a Sender and a Receiver
• Head-End Replication will depend on hardware scale/capability
• Resilient, efficient, and scalable Multicast Forwarding is highly desirable
• Choose the right Multicast Routing Protocol for your need (type/mode)
• Use redundant Multicast Rendezvous Points (Spine/Aggregation generally preferred)
• 99% percent of Overlay problems are in the Underlay (OTV experience)
Keep in Mind
Overlay Convergence = Underlay Convergence!
Underlay – Ingress Replication
• A Packet Multiplication
• EVPN assists no Peer, VNI Topology
• Various Platform Support

• Nexus 9000 Underlay
• Ingress Replication
•
Host sends 1 Packet to Edge-Device
• Edge-Device Encapsulates 1 Packet
and multiplies it
• Ingress VTEP sends 1 Packet per
Neighbor
Overlay Design
Review
Overlay
• EVPN MP-BGP Primer

• Overlay Configuration Steps
1. Define the VTEP Interface
2. Establish the MP-BGP Control Plane
3. Define VLAN-VXLAN Mapping
4. VXLAN Routing Configuration
5. VXLAN HW Gateway Redundancy
6. Connect the Fabric to External Networks
EVPN MP-BGP Primer (1)
▪ Virtual Routing and Forwarding (VRF) VRF Info Name:

VRF Info
VRF-A
Name: VRF-A RD: 50000:1.1.1.2 (auto)
Layer-3 segmentation for tenants’ routing space RD: Imp Route-Target
RR
50000:1.1.1.1 (auto) RR 65000:50000 (auto)
Imp Route-Target 65000:50000 Exp Route-Target 65500:50000 (auto)
VRF Info (auto)
▪ Route Distinguisher (RD): Exp Route-Target 65500:50000
Name: (auto)VRF-A
RD: 50000:1.1.1.3 (auto)
8-byte field, VRF parameters; unique value to
Imp Route-Target 65000:50000 (auto)
make VPN IP routes unique: RD + VPN IP prefix Exp Route-Target 65500:50000 (auto)
▪ Selective distribute VPN routes: V1 V2

Route Target (RT): 8-byte field, VRF parameter,
unique value to define the import/export rules for
VPNv4 routes
BGP Route-Reflector
▪ VPN Address-Family:
RR
Distribute the MP-BGP VPN routes

V3 iBGP Adjacency
▪ Virtual Routing and Forwarding (VRF)

Layer-3 segmentation for tenants’ routing space RR RR
▪ Route Distinguisher (RD):

make VPN IP routes unique: RD + VPN IP prefix
▪ Selective distribute VPN routes: V1 V2

VPNv4 routes
▪ BGP Route-Reflector
RR

V3 ▪ iBGP Adjacency
▪ Virtual Routing and Forwarding (VRF)


▪ MAC_A / IP_A >> LOCAL
▪ Selective distribute VPN routes: ▪ Route-Type2

V 1 V2
VPNv4 routes
RR

▪ Host A
▪ MAC_A / IP_A
▪ BGP Advertisement
▪ VPN-EVPN: RD:[MAC_A][IP_A]
▪ BGP Next-Hop: V1
▪ Virtual Routing and Forwarding (VRF) ▪ Route Target: 65500:50000
▪ Label: 50000


V 1 V2
VPNv4 routes
RR

▪ Host A
▪ MAC_A / IP_A
▪ BGP Advertisement
▪ VPN-EVPN: RD:[MAC_A][IP_A]
▪ BGP Next-Hop: V1
▪ Virtual Routing and Forwarding (VRF) ▪ Route Target: 65500:50000
▪ Label: 50000


V 1 V2
▪ MAC_A / IP_A >> V1
▪ Route-Type2
VPNv4 routes
RR

▪ Host A
▪ MAC_A / IP_A
Overlay

1
Define VTEP Interface (VXLAN Tunnel End-Point)
# Features & Globals

feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1)
# Leaf (V1)
interface nve1
RR RR RR RR
source-interface loopback0
host-reachability protocol bgp
iBGP
V1 V2
V3
*Simplified BGP configuration; would have 4 BGP peers (RR)
IGP not shown
1
# Features & Globals Enables VTEP support (only required on Leaf or Border)
feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1)
# Leaf (V1)
interface nve1
RR RR RR RR
iBGP
V1 V2
V3
IGP not shown
1
feature bgp
feature nv overlay
nv overlay evpn
Enables EVPN Control-Plane in BGP
# Spine (S1)
# Leaf (V1)
interface nve1
RR RR RR RR
iBGP
V1 V2
V3
IGP not shown
1
feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1)
# Leaf (V1) Configure the VTEP interface

interface nve1
RR RR RR RR
iBGP
V1 V2
V3
IGP not shown
1
feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1)

interface nve1
RR RR RR RR
source-interface loopback0 Use a Loopback for Source Interface
iBGP
V1 V2
V3
IGP not shown
1
feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1)

interface nve1
RR RR RR RR
source-interface loopback0 Use a Loopback for Source Interface
iBGP
V2
Enable BGP for Host reachabilityV
1
V3
IGP not shown
Overlay

2
Building the Overlay Control-Plane
feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1)
router bgp 65500
router-id 10.10.10.S1
address-family ipv4 unicast
RR RR RR RR
neighbor 10.10.10.0/24 remote-as 65500
update-source loopback0
address-family l2vpn evpn
send-community both iBGP
route-reflector-client V1 V2
# Leaf (V1)
router bgp 65500
router-id 10.10.10.V1
neighbor 10.10.10.S1 remote-as 65500 V3
update-source loopback0 *Simplified BGP configuration; would have 4 BGP peers (RR)
address-family l2vpn evpn IGP not shown
send-community both
*
2
feature bgp
feature nv overlay
nv overlay evpn
# Spine (S1) Dynamic BGP neighbor

router bgp 65500
RR RR RR RR
update-source loopback0
# Leaf (V1)
router bgp 65500
send-community both
*
2
feature bgp
feature nv overlay
nv overlay evpn

router bgp 65500
RR RR RR RR
update-source loopback0 Activate L2VPN EVPN under each BGP neighbor
# Leaf (V1)
router bgp 65500
send-community both
*
2
feature bgp
feature nv overlay
nv overlay evpn

router bgp 65500
RR RR RR RR
update-source loopback0 Activate L2VPN EVPN under each BGP neighbor
# Leaf (V1)
router bgp 65500
router-id 10.10.10.V1 Send Extended BGP Community
address-family ipv4 unicast to distribute EVPN route attributes V3
neighbor 10.10.10.S1 remote-as 65500
send-community both
*
Overlay

Logical Construct of Multi-Tenant VXLAN EVPN
Tenant A (VRF A)
Tenant A (VRF A)
VLAN A VLAN B VLAN N
Tenant A (VRF A)
SVI SVI SVI
A B N
Tenant A (VRF A)
SVI SVI SVI
A B N
Layer-2 VNI A’ Layer-2 VNI B’ Layer-2 VNI N’
Tenant A (VRF A)
SVI SVI SVI
A B N
Layer-2 VNI A’ Layer-2 VNI B’ Layer-2 VNI N’
• One VLAN maps to one Layer-2 VNI per Layer-2 segment

• A Tenant can have multiple VLANs, therefore multiple
Layer-2 VNIs
• Traffic within one Layer-2 VNI is bridged
• Traffic between Layer-2 VNI’s is routed
Tenant A (VRF A)
SVI SVI SVI SVI
A B N X
VLAN A Layer-3 VNI X’

VLAN B VLAN N
Layer-2 VNI A’ Layer-2 VNI B’ Layer-2 VNI N’ VLAN X
• One VLAN maps to one Layer-2 VNI per Layer-2 segment • 1 Layer-3 VNI per Tenant
• A Tenant can have multiple VLANs, therefore multiple (VRF) for routing
Layer-2 VNIs • VNI X’ is used for routed
• Traffic within one Layer-2 VNI is bridged packets
• Traffic between Layer-2 VNI’s is routed
3
Define VLAN-VXLAN Mapping
Example VLAN Based CLI
# Features
• CLI offers a simplified method of mapping feature vn-segment-vlan-based
a 802.1Q VLAN ID to a VXLAN VNI # VLAN to VNI mapping

vlan 43
• VLAN to VNI configuration on a per-Switch based vn-segment 30000
• VLAN becomes “Switch Local Identifier”
# Activate Layer-2 VNI for EVPN
• VNI becomes “Network Global Identifier” evpn
vni 30000 l2
• 4k VLAN limitation per-Switch still applies rd auto
• 4k Network limitation across fabric has been removed route-target import auto
route-target export auto
• Dependent on VLAN Space!
# Activate Layer-2 VNI on VTEP
interface nve1
Bridge- member vni 30000
Domain
mcast-group 239.239.239.100
suppress-arp
ethernet VLAN VNI vxlan
Multi-Tenancy Full (MT-Full)
3
# Features

vlan 43 VLAN to Layer-2 VNI mapping
vni 30000 l2
• 4k VLAN limitation per-Switch still applies rd auto
interface nve1
Domain
mcast-group 239.239.239.100
suppress-arp
3
# Features

vni 30000 l2 Enables EVPN
• 4k VLAN limitation per-Switch still applies rd auto Control-Plane for
Layer-2 Services
interface nve1
Domain
mcast-group 239.239.239.100
suppress-arp
3
# Features

vni 30000 l2 Enables EVPN
• 4k VLAN limitation per-Switch still applies rd auto Control-Plane for
Layer-2 Services
interface nve1
Bridge-
Domain
member vni 30000 Enables Layer-2
mcast-group 239.239.239.100 VNI on VTEP and
suppress-arp Suppress ARP
3
Port-Local VXLAN-VXLAN Mapping
vlan 3100
vn-segment 31000
▪ Available on N9K from 7.0(3)l1(2) Release vlan 3101
vn-segment 31001
▪ Allows to map the same 802.1Q VLAN tag to vlan 3102
different VNIs on different interfaces of the same leaf vn-segment 31002
node vlan 3103
vn-segment 31003
▪ Current limit is 100 PV mappings per interface, and !
total 1K L2 VNIs per leaf interface Ethernet1/7
switchport mode trunk
VXLAN switchport vlan mapping enable
Underlay
switchport vlan mapping 3000 3100
switchport trunk allowed vlan 3100,3101
!
interface Ethernet1/8
switchport vlan mapping enable
switchport trunk allowed vlan 3102-3103
3001
3000
3001 3000
3
Example BD Based CLI
# VLAN to VNI mapping

• Configuration is extended to allow Per-Port vni 30000
VLAN to Bridge-Domain mapping. Bridge-
bridge-domain 100
Domain will be mapped to VXLAN VNI member vni 30000
• VLAN to VNI configuration on a per-Port based
encapsulation profile vni MyProfile
• VLAN becomes “Port Local Identifier” dot1q 43 vni 30000
• Bridge-Domain becomes “Switch Local Identifier”
# Interface Configuration
• VNI becomes “Network Global Identifier” interface Ethernet 1/12
• 4k VLAN limitation resides only on a per-Dot1Q Trunk no switchport
service instance 1 vni
• Independent from VLAN Space! encapsulation profile MyProfile default
Bridge-
Domain
3
Example BD Based CLI
# VLAN to VNI mapping

• Configuration is extended to allow Per-Port vni 30000
VLAN to Bridge-Domain mapping. Bridge-
bridge-domain 100
Domain will be mapped to VXLAN VNI VLAN to Bridge Domain mapping
member vni 30000
• VLAN to VNI configuration on a per-Port based
encapsulation profile vni MyProfile
• VLAN becomes “Port Local Identifier” dot1q 43 vni 30000
• Bridge-Domain becomes “Switch Local Identifier”
• VNI becomes “Network Global Identifier” interface Ethernet 1/12
• 4k VLAN limitation resides only on a per-Dot1Q Trunk no switchport
service instance 1 vni
• Independent from VLAN Space! encapsulation profile MyProfile default
Bridge-
Domain
Overlay

Gateway Functions in VXLAN
VXLAN Routing
VY
VX
V2 V2
V3 V3
V1 V1
Centralized Gateway Distributed Gateway

• Extra Bridging hop before and after Routing
• Route or Bridge at Leaf
• Distributed Gateway (Anycast) for Routing
• Centralized Gateway (Aggregation) for Routing
• Disaggregate state by scale out
• Large amounts of state => convergence issues
• Optimal Scalability
• Scale problem for large Layer-2 domains
• Requires VXLAN/EVPN!
• Works with VXLAN Flood & Learn or EVPN
VXLAN Routing
Layer-3 Boundary
VY
VX
V2 V2
V3 V3
V1 V1

VXLAN Routing
Layer-3 Boundary
VY
VX
V2 V2
V3 Layer-3 Boundary V3
V1 V1

Centralized Gateway (FHRP) *
VXLAN Routing
• Centralized Routing in a Layer-2 VXLAN VY

Network
• Routing between VNI ( Different Subnet)
VX
• Bridging within VNI (Same Subnet)
• Inter-VXLAN Routing at V2
Core/Aggregation Layer
• vPC provides MAC state V3
synchronization and HSRP peering
• Redundant VTEPs share Anycast VTEP IP address
in the Underlay V1
*Only Flood&Learn Host Y

VNI 30001
Host A
VNI 30000
Distributed IP Anycast Gateway*
VXLAN/EVPN
• Distributed Routing with IP Anycast

Gateway (Integrated Route/Bridge IRB)
• Routing between VNI (Different Subnet)
• Bridging within VNI (Same Subnet)
V2
• Inter-VXLAN Routing Leaf/Access Layer
• All Leafs share gateway IP and MAC for a Subnet (No V3
HSRP)
• A Host will always find its Gateway directly attached
anywhere it moves
V1
Host Y
VNI 30001
*Requires EVPN Control-Plane.
Host A
VNI 30000
Different Integrated Route/Bridge (IRB) Modes
▪ Overlay Networks do follow two slightly

different integrated Route/Bridge (IRB)
?
semantics
▪ Asymmetric V2
• Route and Bridge on the ingress VTEP
• Bridge on the egress VTEP V3
▪ Symmetric
• Route on both the ingress and egress V1
VTEPs
▪ Cisco follows Symmetric IRB Host Y

VLAN 55
Host A
VLAN 43
Routing in VXLAN
▪ VNI utilized for providing isolation at Layer-

2 and Layer-3 across VXLAN
• Received frames must be mapped to specific
VNI for VXLAN transport V2
• The VLAN-to-VNI mapping is performed on
Routing
V3
▪ All Routed Traffic uses the VNI assigned to
the VRF
VLAN
V1
Host Y
VLAN VNI 30001
Host A
VNI 30000
Routing in VXLAN
▪ VNI utilized for providing isolation at Layer-

2 and Layer-3 across VXLAN
• Received frames must be mapped to specific
VNI for VXLAN transport V2
• The VLAN-to-VNI mapping is performed on
Routing
V3
▪ All Routed Traffic uses the VNI assigned to
the VRF
VLAN
V1
Host Y
VLAN VNI 30001
Host A
VNI 30000
Asymmetric IRB
▪ Asymmetric
• Similar to Inter-VLAN routing
• Source and Destination VNIs have to exist
on leaf nodes where routing happens V2
• Post Routing traffic shares destination VNI
with Bridged traffic V3
• Not very suitable for distributed Routing
From Host A via VLAN/VNI “blue” routed
at V1 to VNI “red” reaching destination V1
VLAN “red”
From Host Y via VLAN/VNI “red” routed at Host Y
VLAN 55
V3 to VNI “blue reaching destination VLAN Host A
“blue” VLAN 43
Asymmetric IRB
▪ Asymmetric
VLAN “red”
VLAN 55
“blue” VLAN 43
Asymmetric IRB
▪ Asymmetric
VLAN “red”
VLAN 55
“blue” VLAN 43
Symmetric IRB
▪ Symmetric
• Similar to creating a Transit Segment
• Regardless of where Source or Destination
VNIs exist V2
• Post Routing traffic uses different VNI than
Bridged traffic V3
• Additional VNI for Routing traffic (per VRF)
From Host A via VLAN “blue” routed at V1
to VNI “purple” reaching destination VLAN V1
“red”
From Host Y via VLAN “red” routed at V3 to Host Y
VLAN 55
VNI “purple” reaching destination VLAN Host A
“blue” VLAN 43
• Used in Cisco VXLAN/EVPN
Symmetric IRB
▪ Symmetric
VNIs exist V2
Bridged traffic V3
“red”
VLAN 55
“blue” VLAN 43
Symmetric IRB
▪ Symmetric
VNIs exist V2
Bridged traffic V3
“red”
VLAN 55
“blue” VLAN 43
Host Subnet Redistribution
▪ Host “A” is a silent Host

• Not known via ARP/IP
▪ How can Host “Y” reach Host “A”

V2
• Host “A” and “Y” are in different
VLAN/Subnet
▪ Route for Host “A”-Subnet will be V3

advertised by V1 and V2
▪ Host “Y” will reach either V1 or V2 based V1
on ECMP
Host Y
▪ From V1
or V 2,
Host “A” can be reached VLAN 55
via Layer-2 Segment

Host A
VLAN 43
Host Subnet Redistribution
▪ Host “A” is a silent Host

I know Subnet “A”
• Not known via ARP/IP
▪ How can Host “Y” reach Host “A”

V2
• Host “A” and “Y” are in different
VLAN/Subnet
▪ Route for Host “A”-Subnet will be V3

advertised by V1 and V2
▪ Host “Y” will reach either V1 or V2 based V1
on ECMP
Host Y
▪ From V1
or V 2,
Host “A” can be reached VLAN 55
via Layer-2 Segment

Host A
VLAN 43
Asymmetric vs. Symmetric IRB
VXLAN Routing Block Diagram
• Asymmetric IRB ▪ Symmetric IRB

V1 V3 V1 V3

Layer-2 VNI
V1 V3 V1 V3
Layer-2 VNI
Layer-2 VNI

Layer-2 VNI VRF VNI
V1 V3 V1 V3
Layer-2 VNI
Layer-2 VNI
Layer-2 VNI
Layer-2 VNI
4
Routing in VXLAN – VRF Routing Instance
# VLAN to L3 VNI Mapping
VNI 50000 vxlan

vlan 2500
vn-segment 50000
# Define SVI for VRF routing instance

interface Vlan2500
no shutdown
mtu 9216
vrf member VRF-A
ip forward
# VRF configuration for “customer” VRF

VRF-A vrf context VRF-A
vni 50000
rd auto
ethernet ethernet route-target both auto
route-target both auto evpn

interface nve1
member vni 50000 associate-vrf
4
VNI 50000 vxlan

vlan 2500
vn-segment 50000 VLAN to Layer-3 VNI mapping

interface Vlan2500
no shutdown
mtu 9216
vrf member VRF-A
ip forward

vni 50000
rd auto

interface nve1
4
VNI 50000 vxlan

vlan 2500

interface Vlan2500
VLAN to Layer-3 VNI mapping
no shutdown
mtu 9216 - ip forward required
vrf member VRF-A
ip forward

vni 50000
rd auto

interface nve1
4
VNI 50000 vxlan

vlan 2500

interface Vlan2500
no shutdown
vrf member VRF-A
ip forward

vni 50000
VRF context definition
rd auto - VNI
route-target both auto
- Route-Distinguisher
ethernet ethernet
route-target both auto evpn - Route-Targets
- IPv4 and/or IPv6
interface nve1
4
VNI 50000 vxlan

vlan 2500

interface Vlan2500
no shutdown
vrf member VRF-A
ip forward

vni 50000
VRF context definition
rd auto - VNI
route-target both auto
- Route-Distinguisher
ethernet ethernet
route-target both auto evpn - Route-Targets
- IPv4 and/or IPv6
Enables Layer-3 VNI on VTEP interface nve1
and associate it to VRF (one entry member vni 50000 associate-vrf
per tenant/VRF)
4
Enable Distributed IP Anycast Gateway*
Configuration Example for “BLUE” (V1 & V3) Configuration Example for “RED” (V1-3)
# Features # Features
feature interface-vlan feature interface-vlan
# VLAN to L2 VNI mapping (MT-Lite) # VLAN to L2 VNI mapping (MT-Lite)

vlan 43 vlan 55
vn-segment 30000 vn-segment 30001
# Anycast Gateway MAC, inherited by any interface # Anycast Gateway MAC, inherited by any interface
(SVI) using “fabric forwarding” (SVI) using “fabric forwarding”
fabric forwarding anycast-gateway-mac fabric forwarding anycast-gateway-mac
0002.0002.0002 0002.0002.0002
# Distributed IP Anycast Gateway (SVI) # Distributed IP Anycast Gateway (SVI)

interface vlan 43 interface vlan 55
no shutdown no shutdown
vrf member VRF-A vrf member VRF-A
ip address 11.11.11.1/24 tag 12345 ip address 98.98.98.1/24 tag 12345
fabric forwarding mode anycast-gateway fabric forwarding mode anycast-gateway
*Requires EVPN Control-Plane. VRF and BGP configuration not shown
4
Routing in VXLAN – Advertise Local IP Subnets
VNI 50000 vxlan

# Route-Map for Redistribute Subnet
route-map REDIST-SUBNET permit 10
match tag 12345
# Control-Plane configuration for VRF (Tenant)

router bgp 65500
…
vrf VRF-A
advertise l2vpn evpn
VRF-A redistribute direct route-map REDIST-SUBNET
maximum-paths ibgp 2
ethernet ethernet
4
Routing in VXLAN – Advertise Local IP Subnets
VNI 50000 vxlan

# Route-Map for Redistribute Subnet
route-map REDIST-SUBNET permit 10
match tag 12345
# Control-Plane configuration for VRF (Tenant)

router bgp 65500
…
vrf VRF-A
VRF/Tenant definition advertise l2vpn evpn
within
VRF-AOverlay Control-Plane redistribute direct route-map REDIST-SUBNET
maximum-paths ibgp 2
ethernet ethernet
Overlay

5
VXLAN HW Gateway Redundancy (vPC)
Southbound Connectivity
• VXLAN vPC Domain Configuration

Classic Ethernet
• Configure VXLAN specific vPC Peer-
Link Configuration
• Extend the IP Interface (Loopback) V5
configuration for the VTEP
V4
• Secondary IP address (Anycast) is used as the
Anycast VTEP address
• Both vPC VTEP switches need to have the identical
secondary IP address configured under the
loopback interface
Host D
VNI 30000
5
# VLAN to VNI mapping (MT-Lite)

vlan 55
vn-segment 30000 interface loopback0
ip address 10.10.10.5/32
ip address 10.10.10.99/32 secondary
# VTEP IP Interface; Source/Destination for all
VXLAN Encapsulated Traffic.
▪ Primary IP address is used for Orphan Hosts
▪ Secondary IP is for vPC Hosts (same IP on both
vPC Peers)
interface loopback0 V5
ip address 10.10.10.5/32
V4
# VTEP configuration using Loopback as source.
Destination Group for VNI 30001 is “239.1.1.2”
interface nve1
member vni 30000
mcast-group 239.239.239.100 Host D
suppress-arp VNI 30000
5
# VLAN to VNI mapping (MT-Lite)

vlan 55
vn-segment 30000 interface loopback0
ip address 10.10.10.5/32
# VTEP IP Interface; Source/Destination for all
VXLAN Encapsulated Traffic.
▪ Primary IP address is used for Orphan Hosts
▪ Secondary IP is for vPC Hosts (same IP on both
vPC Peers)
interface loopback0 Add Secondary IP to VTEP Loopback V5
ip address 10.10.10.5/32
V4
# VTEP configuration using Loopback as source.
Destination Group for VNI 30001 is “239.1.1.2”
interface nve1
member vni 30000
mcast-group 239.239.239.100 Host D
suppress-arp VNI 30000
5
Do Not Forget!
# VPC Domain Configuration

vpc domain 99 interface loopback0
peer-switch ip address 10.10.10.5/32
peer-keepalive destination V4-mgmt source v5-mgmt
peer-gateway
ip arp synchronize
# VPC Peer-Link
interface port-channelXX V5
vpc peer-link
V4
# VPC Domain Routing Adjacency*
interface Vlan3999
no shutdown
ip address 10.254.254.1/30
interface loopback0
ip router ospf 1 area 0.0.0.0 ip address 10.10.10.4/32
ip ospf network point-to-point ip address 10.10.10.99/32 secondary
ip pim sparse-mode Host D
VNI 30000
*Best practice on Border Leaf nodes is to enable peering also for each defined VRF (on dedicated SVI interfaces)
5
Do Not Forget!

peer-gateway
ip arp synchronize
# VPC Peer-Link
vpc peer-link
V4
interface Vlan3999
no shutdown
ip address 10.254.254.1/30
interface loopback0
VNI 30000
5
Do Not Forget!

peer-gateway
ip arp synchronize
# VPC Peer-Link
vpc peer-link
V4
Routed Interface (SVI) for routing
interface Vlan3999 adjacency across VPC Peer-Link
no shutdown
ip address 10.254.254.1/30
interface loopback0
VNI 30000
Overlay

6
VXLAN/EVPN Fabric External Routing
▪ The Border Leaf/Spine provides Layer-

2 and Layer-3 connectivity to external
networks
▪ Flexible routing protocol options for
external routing V2
▪ Today, VRF-lite allows to extend VRFs

V3
outside of the fabric
VBL
▪ With Nexus 7000/7700 and F3, LISP
becomes available for fabric extension V1
WAN
6
VBL
VRF VRF VRF V2
A B C
V3
V1
WAN
6
VRFs for External Routing

need to exist on Border Leaf
VBL
VRF VRF VRF V2
A B C
V3
V1
WAN
6

VBL
VRF VRF VRF V2
A B C
V3
Interface-Type Options:
• Physical Routed Ports V1
• Sub-Interfaces
• VLAN SVIs over Trunk Ports
WAN
6

VBL
VRF VRF VRF V2
A B C
V3
Interface-Type Options:
• Physical Routed Ports V1
• Sub-Interfaces
• VLAN SVIs over Trunk Ports Peering Interface can
be in Global or Tenant VRF
WAN
6
VXLAN/EVPN Fabric External Routing (eBGP)
VXLAN Fabric Side Configuration
# Sub-Interface Configuration
no switchport
interface Ethernet1/1.10
mtu 9216
VBL
VRF VRF VRF V2
encapsulation dot1q 10 A B C
vrf member VRF-A
ip address 10.254.254.1/30 V3
# eBGP Configuration
router bgp 65500
…
vrf VRF-A
V1
aggregate-address 10.0.0.0/8 summary-only
neighbor 10.254.254.2 remote-as 65599
update-source Ethernet1/1.10
WAN
Ensure that non-necessary routes are not advertised
towards the External Network
AS# 65599
6
no switchport
mtu 9216
VBL
VRF VRF VRF V2
vrf member VRF-A
ip address 10.254.254.1/30 V3
router bgp 65500
…
vrf VRF-A
Advertise external learned routes V1
address-family ipv4 unicast into EVPN (Route-Type 5)
WAN
AS# 65599
6
no switchport
mtu 9216
VBL
VRF VRF VRF V2
vrf member VRF-A
ip address 10.254.254.1/30 V3
router bgp 65500
…
vrf VRF-A
Advertise external learned routes V1
WAN
Advertise an aggregate of the internal prefixes
AS# 65599
6
VBL
VRF VRF VRF V2
WAN Router Side Configuration A B C
V3
vrf member VRF-A
ip address 10.254.254.2/30
# eBGP Configuration V1
router bgp 65599
…
vrf VRF-A
update-source Ethernet1/1
address-family ipv4 unicast WAN
AS# 65599
6
VXLAN/EVPN Fabric External Routing (OSPF)
no switchport
interface Ethernet1/1.10 VBL
VRF VRF VRF V2
mtu 9216
vrf member VRF-A
ip address 10.254.254.1/30 V3
ip router ospf 1 area 0.0.0.0
ip ospf network point-to-point
# BGP Configuration V1
router bgp 65500
…
retain route-target all
vrf VRF-A
redistribute bgp 100 route-map OSPF-BGP* WAN
6
no switchport
VRF VRF VRF V2
mtu 9216
vrf member VRF-A
ip address 10.254.254.1/30 V3
router bgp 65500
…
retain route-target all Advertise external learned routes
vrf VRF-A
6
no switchport
VRF VRF VRF V2
mtu 9216
vrf member VRF-A
ip address 10.254.254.1/30 V3
router bgp 65500
…
retain route-target all Advertise external learned routes
vrf VRF-A
towards the External Network Redistribute internal prefixes with route-map
Migrating to
VXLAN EVPN
Converting from vPC
VXLAN Design Considerations
VXLAN Mode:
• Flood-and-Learn
• With EVPN control Plane
VXLAN Mode:
• Flood-and-Learn
BUM Traffic Handling:

• Multicast replication
• Unicast/ingress replication
VXLAN Mode:
• Flood-and-Learn

Deployment Scenarios:
• Brown field vs green field
• Investment protection
• Multi-vendor environment?
VXLAN Mode: Scalability:

• Flood-and-Learn • The number of VXLAN VNIs
• With EVPN control Plane • The number of VTEP peers
• The number of EVPN tenants
• The number of VXLAN Host IP routes
• The number of VXLAN Host MAC addresses
• The number of IPv4/IPv6 LPM routes
Deployment Scenarios: • The number of Ingress replication peers
• Brown field vs green field
• Investment protection
• Multi-vendor environment?
VXLAN EVPN Loop Avoidance Considerations
Layer 2
Domain
VXLAN EVPN Loop Avoidance – Option 1
Single logical
connection to the
external L2 domain
Layer 2
Domain
Add BPDU-Guard on all the server facing interfaces
VXLAN EVPN Loop Avoidance – Option 2
Single logical
connection to the
external L2 domain
Layer 2
Domain
Add BPDU-Guard on all the server facing interfaces
Starting Point – Brownfield Network (vPC Based)
WAN - Core ▪ Starting from a traditional network (named

‘Brownfield’)
▪ Brownfield network could be built
L3 leveraging vPC, STP, FabricPath or even
L2 3rd party devices
Migration considerations equally apply to all
those cases
▪ Assumption is also that network services
are connected in traditional fashion (routed
mode at the aggregation layer)
vPC Based Brownfield Network
And What About FabricPath?
WAN - Core ▪ Starting from a traditional network (named

‘Brownfield’)
▪ Brownfield network could be built
L3 leveraging vPC, STP, FabricPath or even
L2 3rd party devices
Migration considerations equally apply to all
those cases
▪ Assumption is also that network services
are connected in traditional fashion (routed
mode at the aggregation layer)
vPC Based Brownfield Network
Deployment
Building Small Initial VXLAN EVPN POD
The end goal is to migrate endpoints and network services to the ACI fabric
WAN - Core
Greenfield VXLAN EVPN Fabric

Brownfield Network
Integration
Connecting Brownfield and Greenfield Networks
First step: creating a L2 connectivity path
WAN - Core
Back-to-back vPC for

L3 avoiding L2 loops
L2
L2 Trunk
Endpoints Integration
Mapping VLANs to L2 VNIs
WAN - Core
Map VLAN10/L2 VNI1

and VLAN20/L2 VNI2
L3
L2
VLAN 10 VLAN 10 VLAN 20 VLAN 20 VLANs 10, 20
App1 Web App1 Web App2 Web Greenfield VXLAN EVPN Fabric
App2 Web 10.20.20.11
10.10.10.10 10.10.10.11 10.20.20.10
▪ Endpoints connected to different VLANs in the brownfield network

▪ Each legacy VLAN is trunked to the VXLAN fabric and mapped to a dedicated L2 VNI
Use Case 1: VLAN == VNI
WAN - Core Greenfield VXLAN EVPN Fabric
Map VLAN10/L2 VNI1

and VLAN20/L2 VNI2
L3
L2
VLAN 10 VLAN 10 VLAN 20 VLAN 20 VLANs 10, 20
L2
App1 Web App1 Web
Broadcast
10.10.10.10 10.10.10.11 Domain
L2
App1 Web App1 Web Broadcast
10.20.20.10 10.20.20.11 Domain
Use Case 1: VLAN == VNI
Map VLAN10/L2 VNI1

and VLAN20/L2 VNI2
L3
L2
VLAN 10
Map VLAN10/L2 VNI1
VLAN 10 VLAN 20 VLAN 20
Web1
App1 Web App1 Web
10.10.10.10 10.10.10.11
Web2
App1 Web App1 Web
10.20.20.10 10.20.20.11
Endpoints Migration
1 - Single VCenter Server Scenario
WAN - Core
L3
L2
Compute
Clusters
100.1.1.3
VM VM
100.1.1.99
VM
100.1.1.7
VM VM
Compute
ClusterBD
VM VM
Existing
VM
VM
App New Compute
Mgmt Cluster Clusters
DVS
vCenter Managed
DVS
Endpoints Migration
1.1 Connect the new ESXi hosts to the vCenter managed DVS
WAN - Core
L3
L2
100.1.1.3
VM VM
100.1.1.99
VM
100.1.1.7
VM VM
BD
VM VM
Existing
VM
VM
App
DVS
Endpoints Migration
1.2 Migrate VMs to the new ESXi cluster

WAN - Core
Migrated VMs still

leverage the gateway
L3 in the Brownfield
network
L2
VM VM
100.1.1.3
VM VM
100.1.1.99 100.1.1.7
VM VM
BD
VM VM
Existing
VM
App
DVS
Endpoints Migration
2 – Multiple VCenter Servers Scenario
WAN - Core
L3
L2
Compute
Clusters
100.1.1.3
VM VM
100.1.1.99
VM
100.1.1.7
VM VM
Compute
ClusterBD Mgmt Cluster
VM VM
Existing vCenter2
VM
VM
App New Compute
Mgmt Cluster Clusters
DVS
Endpoints Migration
2.1 Connect new ESXI servers to a second DVS

WAN - Core
L3
L2
100.1.1.3
VM VM
100.1.1.99
VM
100.1.1.7
VM VM
BD
VM VM
Existing
VM
VM
App
DVS New DVS
Endpoints Migration
2.2 Migrate VMs to the new ESXi cluster*

WAN - Core
Migrated VMs still

leverage the gateway
L3 in the Brownfield
network
L2
VM VM
100.1.1.3
VM VM
100.1.1.99
VM
100.1.1.7
VM VM VM
BD
VM VM
Existing
VM
VM
App
DVS New DVS
*Cross-vCenter vMotion supported with vSphere 6.0

Default Gateway Considerations
Existing Design VXLAN EVPN Fabric
HSRP
Default GW
Subnet 1 = VLAN 10 L2 Bridging Subnet 1 = VNI

VM VM VM
P P P
VM
▪ Default Gateway up to this point is still deployed in the Brownfield network

▪ VXLAN EVPN fabric initially provides only L2 connectivity services
▪ L2 path between the two networks leveraged by migrated hosts to reach the default gateway
Default Gateway Considerations
HSRP
Default GW
Subnet 1 = VLAN 10 L2 Bridging Subnet 1 = VNI

VM VM VM
P P P
VM
▪ Default Gateway up to this point is still deployed in the Brownfield network

▪ VXLAN EVPN fabric initially provides only L2 connectivity services
▪ L2 path between the two networks leveraged by migrated hosts to reach the default gateway
Migrate Default Gateway to the VXLAN Fabric
WAN - Core
Anycast Default
Gateway
L3 L3
L2 L2
VLAN 10 VLAN 20
10.10.10.11 10.20.20.11
10.10.10.10 10.20.20.10
Any IP - Anywhere
Migration
Routing Between Brownfield and Greenfield
▪ Routing between Brownfield and Greenfield may still be required

• Handling communication to IP subnets that remain only on
Brownfield (default gateway remains on aggregation devices)
• Handling communication with the WAN
L3 Routing
HSRP
Default GW
IP Subnet 2 = VLAN 30 IP Subnet 1 = L2 VNI 1

V V V
M M M V V V
P P P P M M M
Migration
Routing Between Brownfield and Greenfield
WAN - Core
Default Gateway for L3 Links
VLAN 30
L3
L2
L3
L2
VLAN 30
10.10.10.11
VLAN 30 NOT carried
on the vPC connection
10.30.30.10
Moving L4-L7 Services
Migrating Network Services
Example of Firewall Services Migration
Starting point:
Active/Standby FW nodes
(routed mode*) connected to
the Aggregation layer
switches
Active Standby
*Similar considerations apply for services in transparent mode
Move the Standby Node to the VXLAN Fabric

Active
FW Keepalives and
state synchronization
Standby
Disconnect the Active Node from the Brownfield Network

Active
Active
FW activated on the
VXLAN fabric
Both Firewall Nodes Connected to the VXLAN Fabric
WAN - Core
Standby Active
Interconnecting
Multiple Sites
VXLAN and DCI
Overlays Evolve and Spread
DC Local Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
DC Local Overlay
End-to-End Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
DC Local Overlay
End-to-End Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
Single Logical Data Center
Changing the Paradigm with Overlays
DC Local Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
DC Local Overlay
Multi-Site Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
DC Local Overlay
Multi-Site Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
Multiple Logical Data Center
VXLAN Evolves as the Control Plane Evolves!
Early Years
Yet Another Encapsulation
▪ Flood & Learn (Multicast-based)
▪ Data-Plane only
Early Years
▪ Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
▪ Control-Plane
▪ Active VTEP Discovery
▪ Multicast and Unicast
Early Years
▪ Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
▪ Control-Plane
▪ Active VTEP Discovery
Today
▪ Multicast and Unicast
VXLAN for DCI – Inter-DC
▪ DCI Ready
▪ ARP/ND caching/suppress
▪ Multi-Homing
▪ Failure Domain Isolation
▪ Loop Protection
Inter-X Connectivity
VXLAN Multi-Pod
EVPN Control-
Fabric #1 BGP EVPN EVPN Control-
Fabric #2
Plane Domain 1 Plane Domain 2
Overlay Overlay
VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P
Bar Bar Bar Bar

em em em em
eta eta eta eta
l l l l
Single Data-Plane – End-to-End
▪ Single Fabric with End-to-

End Encapsulation
▪ Build Hierarchy in the
Underlay – Flatten it in the
Overlay
VXLAN Multi-Pod VXLAN Multi-Fabric
EVPN Control-
Fabric #1 BGP EVPN EVPN Control-
Fabric #2 EVPNFabric
Control-Plane
#1 EVPNFabric
Control-Plane
#2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar
eta eta eta eta em em em em
l l l l etal etal etal etal
Single Data-Plane – End-to-End DCI

Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane
▪ Single Fabric with End-to- ▪ Multiple Fabrics – Normalized

End Encapsulation through Ethernet
▪ Build Hierarchy in the ▪ Multiple Fabrics Interconnect
Underlay – Flatten it in the using DCI (Layer 2 and Layer 3)
Overlay
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site
EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal
Single Data-Plane – End-to-End DCI DCI

Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane
▪ Single Fabric with End-to- ▪ Multiple Fabrics – Normalized ▪ Multiple Fabrics with
End Encapsulation through Ethernet Integrated DCI (DCI2)
▪ Build Hierarchy in the ▪ Multiple Fabrics Interconnect ▪ Integrated DCI – Scaling
Underlay – Flatten it in the using DCI (Layer 2 and Layer 3) within and between
Overlay Fabrics
VXLAN EVPN – Multi-Pod
Underlay Extension
VTEP VTEP VTEP VTEP
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Pod 1 Pod n
Multi-Pod Characteristics – ”The Single”
▪ Single Overlay Domain – End-to-End Encapsulation

▪ Single Overlay Control-Plane Domain – End-to-End EVPN Updates
▪ Single Underlay Domain End-to-End
▪ Single Replication Domain for BUM
▪ Single VNI Administrative Domain
Multi-Pod Characteristics – ”The Single”

▪ Single VNI Administrative Domain
Building Underlay Hierarchies – Non Hierarchical Overlay
Multi-Pod End-to-End Encapsulation
Underlay Extension
VTEP VTEP VTEP VTEP
Spine Spine Spine Spine Overlay Spine Spine Spine Spine
VTEP VTEP
10.1.1.1
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP
Pod 1 Pod n
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
VTEP VTEP
10.1.1.1
10.2.2.7
VTEP
Unicast
Pod 1 Pod n
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
VTEP VTEP
10.1.1.1
10.2.2.7
VTEP
Unicast
Pod 1 Pod n
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
VTEP VTEP
10.1.1.1
10.2.2.7
VTEP
Unicast
Pod 1 Pod n
Baremetal Baremetal
Multi-Pod VXLAN Tunnel Adjacencies
Underlay Extension
VTEP VTEP VTEP VTEP
VTEP VTEP VTEP

10.1.1.1 10.1.1.4 10.2.2.7
Pod 1 Pod n
Multi-Pod VXLAN Tunnel Adjacencies
Underlay Extension
VTEP VTEP VTEP VTEP
VTEP VTEP VTEP

10.1.1.1 10.1.1.4 10.2.2.7
Switch# show nve peers

Interface Peer-IP VNI Up Time
---------- ----------- ------ ----------
nve1 Pod 110.1.1.4 30000 03:18:06 Pod n
nve1 10.2.2.7 30000 00:12:23
Multi-Pod Underlay Extension
Underlay Extension
VTEP VTEP VTEP VTEP
Border (PIP) Border (PIP) Border (PIP) Border (PIP)

10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
VTEP VTEP
10.1.1.1
10.2.2.7
VTEP
Pod 1 Pod 2
Multi-Pod Underlay Extension
POD1 Underlay Routing Table POD2 Underlay Routing Table
Leaf: Border: Leaf: Border: Border: Leaf: Border: Leaf:

10.1.1.1 10.1.1.101 10.2.2.1 10.2.2.101 10.2.2.101 10.2.2.1 10.1.1.101 10.1.1.1
10.1.1.2 10.1.1.102 10.2.2.2 10.2.2.102 10.2.2.102 10.2.2.2 10.1.1.102 10.1.1.2
10.1.1.3 10.2.2.3 10.2.2.3 10.1.1.3
10.1.1.4 10.2.2.4 10.2.2.4 10.1.1.4
10.1.1.5 10.2.2.5 Underlay Extension 10.2.2.5 10.1.1.5
10.1.1.6 10.2.2.6
VTEP VTEP VTEP
10.2.2.6
VTEP
10.1.1.6
10.1.1.7 10.2.2.7 10.2.2.7 10.1.1.7
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
VTEP VTEP
10.1.1.1
10.2.2.7
VTEP
Pod 1 Pod 2
Multi-Pod BUM Replication
Underlay Extension
VTEP VTEP VTEP VTEP
Pod 1 Pod 2
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
BUM
Pod 1 Pod 2
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
BUM
Pod 1 Pod 2
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
BUM
Pod 1 Pod 2
Baremetal
Underlay Extension
VTEP VTEP VTEP VTEP
BUM
Pod 1 Pod 2
Baremetal
Multi-Pod Challenges – ”The Single”

• Scaling the VXLAN EVPN Network

• Overlay Control-Plane Update Propagation

• Network must be extended in Underlay (VTEP to VTEP reachability)

• Network must be extended in Underlay (VTEP to VTEP reachability)
• One BUM flooding domain through out all connected Pods
VXLAN Multi-Site
Functional Components https://tools.ietf.org/html/draft-sharma-multi-site-evpn
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Site 1 Site n
VXLAN Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Site-Internal Fabric
Site 1 (Common VXLAN and Site n
BGP-EVPN Functions)
VXLAN Multi-Site
Site-External DCI
(IP Routing and Increased
MTU Support)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
BGP-EVPN Functions)
VXLAN Multi-Site
Site-External DCI
Border Gateways MTU Support)
(Key Functional Components of
VXLAN Multi-Site Architecture)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
BGP-EVPN Functions)
VXLAN Multi-Site Characteristics
▪ Multiple Overlay Domains – Interconnected & Controlled

▪ Multiple Overlay Control-Plane Domains – Interconnected & Controlled
▪ Multiple Underlay Domains - Isolated
▪ Multiple Replication Domains for BUM – Interconnected & Controlled
▪ Multiple VNI Administrative Domains – Phase 2
VXLAN Multi-Site Characteristics

▪ Multiple VNI Administrative Domains – Phase 2
Underlay Isolation – Overlay Hierarchies
VXLAN Multi-Site
Main Use Cases
VXLAN Multi-Site
Main Use Cases
Scale-Up Model to Build a

Large Intra-DC Network
VXLAN Multi-Site
Main Use Cases

Data Center Interconnect (DCI)
VXLAN Multi-Site
Main Use Cases

Data Center Interconnect (DCI)
Integration with Legacy Networks

(Coexistence and/or Migration)
VXLAN Multi-Site
Underlay Isolation
Multi-Site VIP Multi-Site VIP

10.1.1.111 Site-External DCI 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
VTEP VTEP
10.1.1.1
10.2.2.7
VTEP
Site 1 Site n
VXLAN Multi-Site
Underlay Isolation

VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Site 1 Underlay Site n Underlay

Routing Table Routing Table
Border: Leaf: Border: Leaf:
VTEP 10.1.1.101 10.1.1.1 VTEP
10.2.2.101 10.2.2.1
10.1.1.1
VTEP VTEP VTEP VTEP VTEP
10.1.1.102 10.1.1.2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
10.2.2.102 10.2.2.2
VTEP VTEP
10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

10.1.1.4 10.2.2.4
10.1.1.5 10.2.2.5
Site 1 10.1.1.6 Site n 10.2.2.6
10.1.1.7 10.2.2.7
VXLAN Multi-Site Inter-Site Network
Underlay Isolation Routing Table
Border Site1: Border Site2:
10.1.1.101 10.2.2.101
10.1.1.102 10.2.2.102
10.1.1.111 10.2.2.222

VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Site 1 Underlay Site n Underlay

Routing Table Routing Table
Border: Leaf: Border: Leaf:
VTEP 10.1.1.101 10.1.1.1 VTEP
10.2.2.101 10.2.2.1
10.1.1.1
VTEP VTEP VTEP VTEP VTEP
10.1.1.102 10.1.1.2
10.2.2.7
10.2.2.102 10.2.2.2
VTEP VTEP
10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

10.1.1.4 10.2.2.4
10.1.1.5 10.2.2.5
Site 1 10.1.1.6 Site n 10.2.2.6
10.1.1.7 10.2.2.7
VXLAN Multi-Site
Introducing the Border Gateway
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine
Site 1 Site n
VXLAN Multi-Site
Overlay Multi-Site

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Site 1 Site n
VXLAN Multi-Site
Overlay Multi-Site
Border Gateway (BGW)

- Anycast Cluster -
10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Site 1 Site n
VXLAN Multi-Site
Overlay Multi-Site
Border Gateway (BGW)

- Anycast Cluster -
10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Any VTEP
Site 1 Site n
Multi-Site – VXLAN Tunnel Adjacencies
Overlay Multi-Site

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
VTEP VTEP VTEP

10.1.1.1 VTEP VTEP
10.1.1.4 VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP
VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
Overlay Multi-Site

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
VTEP VTEP VTEP

10.1.1.1 VTEP VTEP
10.2.2.7
VTEP
Leaf1-1# show nve peers

Site 1
---------- ----------- ------ ---------- Site n
nve1 10.1.1.4 30000 03:18:06
nve1 10.1.1.111 30000 00:12:23
BG102# show nve peers
---------- ----------- ------ ----------
nve1
Overlay
10.1.1.1
Multi-Site
30000 00:12:16
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:23

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
VTEP VTEP VTEP

10.1.1.1 VTEP VTEP
10.2.2.7
VTEP

Site 1
---------- ----------- ------ ---------- Site n
nve1 10.1.1.4 30000 03:18:06
nve1 10.1.1.111 30000 00:12:23
BG102# show nve peers
---------- ----------- ------ ----------
nve1
Overlay
10.1.1.1
Multi-Site
30000 00:12:16
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:23

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
VTEP VTEP VTEP

10.1.1.1 VTEP VTEP
10.2.2.7
VTEP

Site 1
---------- ----------- ------ ---------- Site n------
---------- ----------- ----------
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:25
nve1 10.1.1.111 30000 00:12:23
Border Gateway to Cloud
Layer-3
Network
BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Border Gateway Between Spine and Super-Spine
Super-Spine Super-Spine
Spine Spine Spine Spine Spine Spine
Border Gateway on Spine
Super-Spine Super-Spine
Spine Spine Spine Spine Spine Spine Spine Spine Spine Spine
Border Gateway Back-to-Back
BGW BGW BGW BGW

Site 1 Site 2
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
VXLAN Multi-Site
Border Gateways Deployment Considerations
▪ Border Gateways used for two main functions: Anycast Border Gateways
Interconnecting each site to the Inter-Site network (for

BGW BGW BGW BGW
• VTEP VTEP VTEP VTEP
East-West traffic flows)

• Connecting each site to the external Layer 3 domain (for
North-South traffic flows)
• May also be used to connect endpoints and/or network
service nodes (FWs, ADCs) Site 1
▪ Possible deployment models:

VPC Border Gateways
• Anycast Border Gateways (currently supported) BGW BGW
VPC Border Gateways (planned for Q3CY18)

VTEP VTEP
▪ BGW function enablement in the VXLAN EVPN fabric:

• BGWs as leaf nodes
• BGWs as spine nodes (Border-Spines)
Site 1
VXLAN Multi-Site
Anycast Border Gateway (1)
Anycast Border Gateway
▪ Up to 4 Border Gateways
Multi-Site VIP
10.1.1.111 ▪ Border Gateway Support
BGW BGW BGW BGW

▪ Leaf 7.0(3)I7(1), Spine 7.0(3)I7(2)
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4 ▪ Common Multi-Site Virtual IP (Multi-Site VIP)
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
across BGWs
• Multi-Site VIP for communication between the
Multi-Site VIP
10.1.1.111 Border Gateways in different Sites
• Multi-Site VIP for communication between
Border Gateways and Leaf nodes within a Site
▪ Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
Site 1
endpoints (routed only), intra- and inter-Site
VXLAN Multi-Site
Anycast Border Gateway

Type: 00 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:00:07 VNI: 30010 ▪ Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or
BGW BGW BGW BGW
a set of Layer-2 VNIs
VTEP VTEP VTEP VTEP
• DF election and assignment is automatic
▪ Using BGP EVPN Route Type 4 for DF election
DF DF DF DF
30010 30011 30012 30099
• Operator Managed Assignment (Type: 00)

BGP EVPN
• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
RR RR
Spine Spine
• Multi-Site Discriminator (Ethernet-
Segment: 00:00:07)
• Originators IP Address (PIP): 10.1.1.101
Site 1
• Layer-2 VNI: 30010
VXLAN Multi-Site
External
Connectivity Anycast Border Gateway
Point-to-Point L3 Links
(Physical/Sub-Interfaces) ▪ Single-Homed End-Points only connected with
L3 links
• Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW BGW BGW
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4
• External routers
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
• No SVI support on BGW nodes
.1 .1
▪ Advertised and Reachable through Individual
Primary IP Address (PIP)
(Physical/Sub-Interfaces)
ADC ADC
• Intra-Site: Leaf nodes use PIP to reach the device
ADC ADC
connected to Border Gateways
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102 • Inter-Site: Remote Border Gateways use PIP to
reach the device connected to Border Gateways
VTEP
Site 1
Anycast BGW vs. VPC Border Gateway
Anycast Border Gateway VPC Border Gateway

• Up to 4 BGW • 2 BGW with physical VPC Peer-Link
• Shared Nothing • Small Deployments
• Simple Failure Scenarios • End-Point or Network Services
• Any Deployments Connectivity on BGW
• No End-Point or Network Services • Migration Use-Cases (Brownfield)
Connectivity on BGW • Pseudo-BGW to BGW
• Greenfield Deployments • Classic Ethernet/FabricPath to VXLAN
EVPN
VXLAN Multi-Site
VPC Border Gateway and Transit Traffic
VPC Border Gateway

Multi-Site VIP ▪ Common Multi-Site Virtual IP (Multi-Site VIP)
10.1.1.111
across BGWs
BGW BGW • Multi-Site VIP for Inter-Site transit
communication (transit)
VTEP VTEP
PIP-BGW1 PIP-BGW2
10.1.1.101 VPC VIP 10.1.1.102
Common VPC Virtual IP (VPC VIP) across BGWs

10.1.1.121
•
Multi-Site VIP
10.1.1.111
• Used by default for external communication
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
▪ Individual Primary IP (PIP) per BGW
• Used for external communication with
“advertised-pip”
Site 1
VXLAN Multi-Site
VPC Border Gateway and Locally Attached End-Points
VPC Border Gateway

Multi-Site VIP ▪ Single- or Dual-Homed End-Points
10.1.1.111
• Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW
VTEP VTEP • Physical or Virtual Servers
VPC VIP
10.1.1.121 ▪ Advertised and Reachable through VPC Virtual
IP Address (VPC VIP)
Multi-Site VIP
10.1.1.111 • Intra-Site: Leaf nodes use VPC VIP to
ADC Baremetal reach End-Points connected to Border
ADC EP
Gateways
0000.3010.1102 0000.3010.1101
192.168.10.102 192.168.10.101 • Inter-Site: Remote Border Gateways use
VPC VIP to reach End-Points connected to
Border Gateways
• Traffic potentially traverses VPC Peer-Link
Site 1
VXLAN Multi-Site
VPC Border Gateway and Designated BUM Forwarder
VPC Border Gateway

▪ VPC-based Designated Forwarder Election
BGW BGW
VTEP VTEP
▪ Per-Site Designated Forwarder (DF) election
DF VPC VIP
10.1.1.121
• Using same approach as in VPC
• Best Path to Rendezvous-Point or VPC
Primary Node
• Same VPC node is elected DF for all the
Layer-2 VNIs
Site 1
VXLAN Multi-Site
BUM Traffic Forwarding
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Site 1 Site n
VXLAN Multi-Site
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
BUM
Site 1 Site n
Baremetal
VXLAN Multi-Site
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
BUM
Site 1 Site n
Baremetal
VXLAN Multi-Site
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
BUM
Site 1 Site n
Baremetal
VXLAN Multi-Site
BUM Replication Modes (Multicast Intra-Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Multicast Multicast
Site 1 Site n
VXLAN Multi-Site
BUM Replication Modes (Ingress Replication Only)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Ingress Replication
Site 1 Site n
VXLAN Multi-Site
BUM Replication Modes (Mixed Mode Intra-Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Multicast
Site 1 Site n
VXLAN Multi-Site
BUM Traffic Policing
Overlay Multi-Site
Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP
BGW BGW
Unknown Unicast 0-100% BGW BGW
Multicast 0-100%
Spine Spine Spine
BUM
Site 1 Site n
Baremetal
VXLAN Multi-Site
Connectivity to the External Layer 3 Domain
▪ The BGW nodes can also be used to provide Layer-3 external

connectivity to each site
▪ Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers (Shared
Border deployment model, aka GOLF)
• Dedicated or shared pair of WAN Edge routers across sites
▪ External Layer-3 network may be different from the DCI
network used for inter-site communication
VXLAN Multi-Site
Border Gateways and VRF-Lite to External Routers
External
VRF-AVRF-B VRF-C ▪ Separate IPv4/IPv6 routing peering for
Connectivity each VRF established with the
Site-External
external routers on dedicated physical

Multi-Site
Overlay
interfaces/sub-interfaces
▪ Must use separate interfaces for
BGW BGW BGW BGW inter-site communication
No support for VXLAN encapsulated
VTEP VTEP VTEP VTEP
▪
traffic on sub-interfaces
Site-Internal
Site 1
VXLAN Multi-Site
Dedicated physical
interfaces / sub-
interfaces for each VRF External
VRF-AVRF-B VRF-C ▪ Separate IPv4/IPv6 routing peering for
Site-External

Multi-Site
Overlay
VTEP VTEP VTEP VTEP
▪
Site-Internal
Site 1
VXLAN Multi-Site
Dedicated physical Separate IPv4/IPv6 routing

interfaces / sub- peering for each VRF (IGP
interfaces for each VRF External
VRF-AVRF-B VRF-C or eBGP) ▪ Separate IPv4/IPv6 routing peering for
Site-External

Multi-Site
Overlay
VTEP VTEP VTEP VTEP
▪
Site-Internal
Site 1
VXLAN Multi-Site
Border Gateway and Shared Border (aka ‘GOLF’)
▪ Single MP-BGP EVPN peering

External
VRF-A
VRF-B VRF-C
established with the external routers
Connectivity
to exchange routes for all the VRFs
▪ VXLAN Data-Plane between the
Multi-Site BGWs and the external routers
Site-External
Overlay
▪ Same spine uplinks used for all
VXLAN encapsulated traffic (North-
South and East-West)
BGW BGW BGW BGW ▪ Required because of the use of DCI link
VTEP VTEP VTEP VTEP
tracking
Site-Internal
▪ Various northbound hand-off options

depending on specific HW support:
Site 1
VRF-Lite, MPLS-VPN, LISP
VXLAN Multi-Site

External
VRF-A
VRF-B VRF-C
Connectivity
Routed interface extending
‘underlay’ connectivity to ▪ VXLAN Data-Plane between the
the external routers
Site-External
Overlay
VTEP VTEP VTEP VTEP
tracking
Site-Internal

Site 1
VXLAN Multi-Site
External router operates like a

traditional VXLAN EVPN VTEP
(Layer 3 only)
External
VRF-A
VRF-B VRF-C
Connectivity
‘underlay’ connectivity to ▪ VXLAN Data-Plane between the
Site-External
Overlay
VTEP VTEP VTEP VTEP
tracking
Site-Internal

Site 1
VXLAN Multi-Site

(Layer 3 only)
External
VRF-A
VRF-B VRF-C
Connectivity
Single MP-BGP EVPN routing
instance to exchange routes
‘underlay’ connectivity to
for all VRFs
Site-External
Overlay
VTEP VTEP VTEP VTEP
tracking
Site-Internal

Site 1
VXLAN Multi-Site

(Layer 3 only)
External
VRF-A
VRF-B VRF-C
Connectivity
Single MP-BGP EVPN routing
instance to exchange routes
‘underlay’ connectivity to
for all VRFs
Site-External
Overlay
BGW BGW BGW BGW
VXLAN Data Plane
▪ Required because of the use of DCI link
VTEP VTEP VTEP VTEP
between BGW and WAN tracking
Edge Router
Site-Internal

Site 1
VXLAN Multi-Site
Legacy Site Integration
BGW
VTEP VTEP VTEP VTEP
BGW BGW
Spine Spine Spine Spine Baremetal ADC
Greenfield Site Legacy Site
▪ Coexistence and/or migration use cases

• Extend Layer-2 and Layer-3 multi-tenant connectivity across sites
▪ Deploy a pair of Pseudo-BGWs in the legacy site
• Simplified configuration required on Pseudo-BGWs nodes
• Still offering native Multi-Site functions (Ingress Replication for BUM, BUM containment, etc.)
VXLAN Multi-Site
Legacy Site Integration
Pair of Pseudo-BGWs
(EX/FX Switches)
BGW
VTEP VTEP VTEP VTEP
BGW BGW

VXLAN Multi-Site
Legacy Site Integration IR for BUM +
aggregated BUM Pair of Pseudo-BGWs
containment (EX/FX Switches)
BGW
VTEP VTEP VTEP VTEP
BGW BGW

Multi-Site and Legacy Site Integration
Default Gateway Deployment – Recommended
BGW
VTEP VTEP L3 VTEP VTEP
BGW BGW
L2
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP
L2
▪ Recommended approach is to migrate the default gateway from the

legacy aggregation devices to the Border Gateways (VXLAN EVPN
Anycast Gateway)
▪ Optimize routing between endpoints deployed across sites
Default Gateway
migrated to the Border
BGW Gateways (VXLAN EVPN
Anycast Gateway)
BGW BGW
L2
L2

Anycast Gateway)
Default Gateway
BGW Gateways (VXLAN EVPN
Anycast Gateway)
BGW BGW
L2
L2
Distributed Anycast Greenfield Site Legacy Site

Gateway function

Anycast Gateway)
Default Gateway
Greenfield VXLAN BGW Gateways (VXLAN EVPN
EVPN Fabric offers L2 VTEP VTEP L3 VTEP VTEP
Anycast Gateway)
and L3 services for the BGW BGW
stretched IP subnets L2
Legacy
infrastructure offers
only L2 services
L2
Distributed Anycast Greenfield Site Legacy Site

Gateway function

Anycast Gateway)
VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (1)
Pair of Pseudo-BGWs
Pair of Pseudo-BGWs (EX/FX Switches)
(EX/FX Switches) BGW BGW
VTEP VTEP VTEP VTEP
Legacy Site 1 Legacy Site 2
▪ A pair of Pseudo-BGWs inserted in each legacy site to extend Layer-2 and Layer-
3 connectivity between sites
• Replacement of traditional DCI technologies (EoMPLS, VPLS, OTV, …)
▪ Slowly phase out the legacy networks and replace them with VXLAN EVPN fabrics
Convert the nodes to
Convert the nodes to full BGWs functions
full BGWs functions
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
‘Mixed’ Site 1 ‘Mixed’ Site 2
▪ Introduce VXLAN EVPN spines and additional VTEPs in each site

▪ Convert the Pseudo-BGWs to full BGW (may require vPC support on BGWs)
▪ Migrate endpoints between the legacy network and the new VXLAN EVPN fabric
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Greenfield Site 1 Greenfield Site 2
▪ Decommission the legacy networks and leave only the VXLAN EVPN
fabrics in place
VXLAN EVPN – Multi-Site
Multi-Site Core
• Border Gateway (BGW) to Border Gateway (BGW)
reachability required
• Reachability Back-to-Back (full-mesh) or via Layer-3
transport network
• Any Routing Protocol for BG reachability No Underlay Extension
• IPv4 Unicast Transport
VTEP
(Ingress
VTEP
Replication) VTEP VTEP
• BGP full-mesh or Route-Server (eBGP ”Route Reflector”)
for Overlay Control-Plane
Site 1 Site n
VXLAN EVPN – Multi-Site
Multi-Site Core
• Border Gateway (BGW) to Border Gateway (BGW)
reachability required
• Reachability Back-to-Back (full-mesh) or via Layer-3
transport network
• Any Routing Protocol for BG reachability No Underlay Extension
• IPv4 Unicast Transport
VTEP
(Ingress
VTEP
Replication) VTEP VTEP
• BGP full-mesh or Route-Server (eBGP ”Route Reflector”)
for Overlay Control-Plane Multi-Site Border Gateway (BGW):
• Seamless insertion into existing VXLAN EVPN Fabrics
(Border Gateways require Nexus 9x00-EX/-FX)Spine Spine Spine Spine
• Layer-2 and Layer-3 extension to other Sites

• BGP- or VPC-based Border Gateway (BGW) Cluster (up
to 4 nodes when using BGP)
• All Border Gateways (BGW) are representing a common
Anycast VTEP
• Failure containment through Broadcast, Unknown Unicast
and Layer-2 Multicast limiter (off or rate-based)

• Co-Existence with VRF-Lite for External Connectivity
• Core and Fabric link tracking
Site 1 Site n
Multi-Site Advantages – ”The Multiple”

• Scaling and Segregating VXLAN EVPN Networks

• Limited Overlay Control-Plane Update Propagation

• Isolated Underlay Domains – No need for Extension

• Isolated Underlay Domains – No need for Extension
• Individual BUM flooding domain with Traffic control
Multi-Pod Multi-Fabric Multi-Site

Underlay Control Plane Unified Underlay Domain Separated Underlay Domains Separated Underlay Domains
Overlay Control Plane Separated Overlay Control-Plane Domains
Overlay Data Plane Single Data-Plane Separated Data-Planes Separated Data-Planes
Unified Underlay Domain (All

BUM Replication in DCI Dependency on DCI Choice (Unicast/Multicast)
Multicast or All Ingress Replication)
ARP Flood Suppression (DCI) yes yes yes
Unknown Unicast Flood

no yes yes
Suppression (DCI)
Broadcast Suppression/Limit
no yes yes
(DCI)
Layer-2 Loop Prevention Loop mitigation (Edge Protection) VPC at Border Loop mitigation (At DCI)
Virtual Peer Link
(vPC) Update
Traditional vPC Recap
vPC Domain
vPC1 Physical Peer Link vPC2

Orphan Port Orphan Port
Peer Keepalive
vPC
Server Server
Server
vPC for VXLAN and VXLAN EVPN
Spine Spine
Individual Identity vPC1 Individual Identity vPC2

10.1.1.11 10.1.1.12
Anycast VTEP (vip)
10.1.1.10
vPC1 vPC2
Server Server
Server
Spine Spine
Individual Identity vPC1 Individual Identity vPC2

10.1.1.11 10.1.1.12
Anycast VTEP (vip)
10.1.1.10
vPC1 vPC2
EVPN Route Type Attachment Next-hop
Type 5 vPC advertised by vip

(IP Prefix Routes)
Server Orphan advertised by vip Server

(Host Routes) Orphan Port advertised by vip
Server
Spine Spine
Backup Routing over

Peer Link
vPC1 vPC2
Server Server
Server
vPC for VXLAN EVPN
Spine Spine
Per-VRF Peering for IP

Prefix Exchange
vPC1 vPC2
Subnet X Subnet Y
192.168.11.0/24 192.168.12.0/24
vPC for VXLAN EVPN (advertise-pip)
Spine Spine
Individual Identity and Individual Identity and

VTEP vPC1 (pip) VTEP vPC2 (pip)
10.1.1.11 10.1.1.12
Anycast VTEP (vip)
10.1.1.10
vPC1 vPC2
Subnet X Subnet Y
192.168.11.0/24 192.168.12.0/24
Server Server
Server
vPC for VXLAN EVPN (advertise-pip)
Spine Spine
Individual Identity and Individual Identity and

VTEP vPC1 (pip) VTEP vPC2 (pip)
10.1.1.11 10.1.1.12
Anycast VTEP (vip)
10.1.1.10
vPC1 vPC2
Subnet X Subnet Y
192.168.11.0/24 192.168.12.0/24
EVPN Route Type Attachment Next-hop
Type 5 vPC advertised by pip

(IP Prefix Routes)
Server Orphan advertised by pip Server

(Host Routes) Orphan Port advertised by vip
Server
vPC without Peer-Link for (vPC2) VXLAN EVPN
Spine Spine
vPC1 vPC2
Peer Keepalive
vPC
Server Server
Server
Spine Spine
Virtual Peer Link
vPC1 vPC2
Peer Keepalive
vPC
Server Server
Server
Spine Spine
Virtual Peer Link over Fabric (Layer-3)
Virtual Peer Link
• Uses Spines for Redundancy, Resiliency
and Performance
• Doesn’t use VTEP IP address (loopback)
vPC1 vPC2
Peer Keepalive
vPC
Server Server
Server
Spine Spine
Virtual Peer Link over Fabric (Layer-3)
Virtual Peer Link
• Uses Spines for Redundancy, Resiliency
and Performance
• Doesn’t use VTEP IP address (loopback)
vPC1 vPC2
Peer Keepalive
vPC
Server
Peer Keepalive remains Server
• Out-of-Band (mgmt0 or dedicated link)*
• In-Band (dedicated Loopback)
Server
Spine Spine
Virtual Peer Link
vPC1 vPC2
Subnet X Orphan Port Orphan Port Subnet Y

192.168.11.0/24 192.168.12.0/24
Peer Keepalive
vPC
Server Server
Server
• Introduction in NX-OS 9.2(3) • Smaller VTEP Scale per Fabric

• Part of Essentials License • initial release at ~1/3
• “always PIP” mode results in 3 VTEPs
• Supported FX/FX2 Platforms per vPC domain
• EX-based Platform in future • Compensated with upcoming VTEP scale
increase (9.3(x))
• PIM ASM and Ingress-
Replication for BUM • Leaf and Border deployments
only
• PIM BiDir under consideration
• no BGW for Multi-Site support
• TRM Support
• No FEX
VXLAN Tenant
Routed Multicast
(TRM)
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine
RCVR VLAN 101 (Green)

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
TOR3 TOR4
Leaf Leaf
VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
TOR3 TOR4
Leaf Leaf
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
TOR3 TOR4
Leaf Leaf
VXLAN EVPN
S • ”Single Copy” in Core – Treated as BUM

SRC
• Same Subnet Only
TOR1 TOR2
SRC-10 Leaf
• No Pruning on Local
Leaf
Interface or Remote VTEP
224.10.10.10 Interface
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
TOR3 TOR4
Leaf Leaf
VXLAN EVPN
Same Subnet Forwarding with IGMP Snooping
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
S • ”Single Copy” in Core – Treated as BUM

SRC
• Same Subnet Only
TOR1 TOR2
SRC-10 Leaf
• Pruning on Local
Leaf
Interface
224.10.10.10
10.10.10.100 • VXLAN is ”pruned off” if no interest Receiver exists
behind any Remote VTEP
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
Different Subnet Forwarding – Router on-a-Stick
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.10.10.254
10.10.10.100 10.20.20.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.20.20.21
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.10.10.254
10.10.10.100 10.20.20.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.20.20.21
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.10.10.254
10.10.10.100 10.20.20.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.20.20.21
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.10.10.254
10.10.10.100 10.20.20.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.20.20.21
VXLAN EVPN
• Multiple Copy in Core – Treated as BUM

S
SRC
• Different Subnet possible – RPF Challenges
TOR1 TOR2
• Pruning on Local Interface
SRC-10 Leaf Leaf
224.10.10.10 • VXLAN is NOT pruned if interest Receiver exists behind
10.10.10.254
10.10.10.100 one Remote VTEP 10.20.20.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.20.20.21
VXLAN EVPN
Functional Components
Tenant Routed Multicast (TRM)
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR
Baremetal Baremetal Baremetal Baremetal Baremetal
SRC-10 RCVR-10 RCVR-20 RCVR-30 RCVR-11

224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100
https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-seamless-interop
Spine Spine
Site-External DCI
Underlay:
MTU Support)
• PIM-based Underlay Transport (PIM ASM)
• Separate Multicast Groups from Layer-2 VNI
VXLAN EVPN
• Leveraging same redundant Underlay Rendezvous-
Point (i.e. PIM Anycast-RP)
• Single Packet in Core
VTEP VTEP VTEP VTEP
DR DR DR DR
Baremetal Baremetal Baremetal Baremetal Baremetal

224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100
Spine Spine
Site-External DCI
Underlay:
MTU Support)
• PIM-based Underlay Transport (PIM ASM)
• Separate Multicast Groups from Layer-2 VNI
VXLAN EVPN
• Leveraging same redundant Underlay Rendezvous-
Point (i.e. PIM Anycast-RP)
• Single Packet in Core
Overlay:
VTEP VTEP VTEP VTEP
• BGP-based Control-Plane using ngMVPN (Next-
DR DR DR DR
Generation Multicast VPN)
• Using existing BGP Route-Reflector
• Rendezvous-Point-less
• Efficient Single Copy in Multicast Underlay
• Always-Route approach (per-VLAN config)
Baremetal Baremetal Baremetal • Distributed Anycast Designated Router (DR)
Baremetal Baremetal
• VPC – Virtual Port-Channel

• Integration with non-TRM VTEP
224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100
Same Subnet Forwarding w/IGMP Snooping
TRM Forwarding (Layer-2 Only Mode)
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
• Local IGMP Termination (needs Querier)

S
• Bridge approach only – in Layer-2 VNI
SRC
TOR1 • ”Single Copy”TOR2

in Core
SRC-10
• Local and Remote IGMP Snooping
Leaf Leaf
224.10.10.10
10.10.10.100
• Uses BGP EVPN Route-Type 6 (SMET)
R Spine

VLAN 101 (Green)
RCVR-10 Spine
10.10.10.10
R
TOR3 TOR4 RCVR
RCVR-11
Leaf Leaf
10.10.10.11
VXLAN EVPN
Different Subnet Forwarding
TRM Forwarding (Layer-3 Mode)
TOR1 TOR2
Leaf Leaf
Spine
VLAN 101 (Green)

VLAN 101 (Green)
VLAN 202 (Blue)
VLAN 202 (Blue) Spine
TOR3 TOR4
Leaf Leaf
VXLAN EVPN
VLAN 202 (Blue) VLAN 202 (Blue)
Different Subnet Forwarding
TOR1 TOR2
Leaf Leaf
Spine
VLAN 101 (Green)

VLAN 101 (Green)
L3VNI 50001
VLAN 202 (Blue)
VLAN 202 (Blue) Spine
TOR3 TOR4
Leaf Leaf
VXLAN EVPN
VLAN 202 (Blue) VLAN 202 (Blue)
Different and Same Subnet Forwarding
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R
Spine
RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R
Spine
RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R
Spine
L3VNI 50001
RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R
Spine
L3VNI 50001
RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
• Distributed Designated Router (DR)

S
• Always Route approach – in Layer-3 VNI (VRF)
SRC
TOR1 • ”Single Copy”TOR2

in Core
SRC-10
• Egress replication - closest to the fan-out.
Leaf Leaf
224.10.10.10
10.10.10.100
• Single Default MDT (I-PMSI)
R
Spine
L3VNI 50001
RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
Local and Remote Forwarding
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100
R
Spine
L3VNI 50001
RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10
10.10.10.100 TTL Decrement
R TTL Decrement Spine
(routed) L3VNI 50001

RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
Tenant Routed Multicast routes between different IP

S Subnets on the same Switch (Leaf). As a result, the TTL
SRC
is decremented in this routed forwarding operation.
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10

RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN

SRC
TOR1 TOR2
SRC-10 Leaf Local to the sameLeafSwitch (Leaf), Multicast is bridged
224.10.10.10
within the same IP Subnet. During this forwarding
operation, TTL is not decremented.

RCVR
RCVR-20 Spine
10.20.20.20 R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN

SRC
TOR1 TOR2
SRC-10 Leaf Local to the sameLeafSwitch (Leaf), Multicast is bridged
224.10.10.10
within the same IP Subnet. During this forwarding
operation, TTL is not decremented.

Tenant Routed Multicast uses an always route, single
RCVR copy approach across the VXLAN EVPN Fabric. This is
true if Sources and Receivers reside in the same or in
RCVR-20
10.20.20.20
Spine
different IP Subnets. As a result, the TTL is
R
decremented during this forwarding operation.
R TOR3 TOR4 RCVR
All IP Subnets must

Leaf be in the same Routing Domain or
RCVR-21
Leaf
RCVR
Tenant (= VRF) 10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
Overlay Rendezvous Point
• RP-less
S
• Distributed Anycast RP (NGMVPN-based)
SRC
TOR1 • ShortestTOR2
Path Tress (SPT only)
SRC-10
224.10.10.10
Leaf
• Requires per-Tenant
Leaf
Loopback, Multicast
10.10.10.100 TTL Decrement enabled
• External RP

• Centralized RP (PIM-based)
• Shared Tree and Shortest Path Tree (cut over)
RCVR
RCVR-20
10.20.20.20
Spine
• Requires External PIM-based RP
R
R TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
RCVR
10.20.20.21
RCVR-10
10.10.10.10
VXLAN EVPN
Matthias Wessendorf
NX-OS Technical Marketing Engineer
Programmability
Extensibility: Guest
Shell and Docker
Securely Run Custom On-Box Linux Apps
Guest Shell: Secure Linux Container 64

Bit

Bit
CentOS 7.0 rootfs

Bit
Open-Source
Apps
CentOS 7.0 rootfs

Bit
Open-Source Your Custom Apps

Apps (C, Python, Go…)
CentOS 7.0 rootfs
$ dohost
NX-OS Guest Shell: Secure Linux Container 64
JSON Bit
CLI
Apps (C, Python, Go…)
CentOS 7.0 rootfs
$ dohost
JSON Bit
CLI
bootflash: Apps (C, Python, Go…)
CentOS 7.0 rootfs
$ dohost
JSON Bit
CLI
bootflash: Apps (C, Python, Go…)
CentOS 7.0 rootfs
Network
Linux → NX-OS Network Synchronization
mtu 9216
ip address 10.0.1.2/30
no shutdown
[guestshell@guestshell ~]$ ifconfig Eth1-49

interface Ethernet1/49 Eth1-49:
mtu 9216 flags=4163<UP,BROADCAST,RUNNING,MULTICAST>
ip address 10.0.1.2/30 mtu 9216
no shutdown inet 10.0.1.2 netmask 255.255.255.252
broadcast 10.0.1.3

ip address 10.0.1.2/30 mtu 9216
broadcast 10.0.1.3
switch# sh vrf
VRF-Name VRF-ID State
default 1 Up --
management 2 Up --
vpn1 3 Up --

ip address 10.0.1.2/30 mtu 9216
broadcast 10.0.1.3
switch# sh vrf [guestshell@guestshell ~]$ ip netns list

VRF-Name VRF-ID State vpn1
default 1 Up -- management
management 2 Up -- default
vpn1 3 Up -- [guestshell@guestshell ~]$
Richly Populated Repositories for 3rd Party Apps
switch# guestshell
[guestshell@guestshell ~]$ cd /etc/yum.repos.d/
[guestshell@guestshell yum.repos.d]$ ls -l
total 15
-rw-r--r-- 1 root root 1664 Nov 3 19:25 CentOS-Base.repo
-rw-r--r-- 1 root root 1309 Nov 3 19:25 CentOS-CR.repo
-rw-r--r-- 1 root root 649 Nov 3 19:25 CentOS-Debuginfo.repo
-rw-r--r-- 1 root root 1331 Nov 3 19:25 CentOS-Sources.repo
[guestshell@guestshell yum.repos.d]$
[guestshell@guestshell ~]$ chvrf management yum repolist all

Loaded plugins: fastestmirror
[...]
base/7/x86_64 CentOS-7 - Base enabled: 9007
The Guest Shell is Secure
• Namespaces are separate, resource usage is controlled,

access is controlled.

• No visibility into Cisco proprietary software (cannot read,
write, or execute NX-OS binaries).

• No visibility into Cisco proprietary disk partitions.

• No access to internal, Cisco proprietary drivers.

• No access to internal, Cisco proprietary drivers.
• No ability to load kernel drivers.
Linux Apps Can Interact with the External World
Nexus 9K
Your Custom Applications Existing 3rd Party Linux
(Python, C++ etc.) Applications
Guest Shell Linux Networking Stack
NX-OS CLI
L2 L3 Interfaces Platform More…
Docker Engine
Startimg NX-OS 9.2(1) – July 2018
Nexus 9K
Docker Engine
• Available on all Nexus 9K models,

and on Nexus 3K models equipped
with 8G+ of memory.
Nexus 9K
Docker Engine
• Available on all Nexus 9K models,

and on Nexus 3K models equipped
with 8G+ of memory.
• Standard Docker engine with all the

commands supported: Nexus 9K
docker run/pull/push/kill/info
etc.
Standardization, Flexibility, and Efficiency
Guest Shell Docker Engine

Number of container instances One Many
Access to storage and network Yes Yes
Linux distribution type CentOS Any
Container manipulation NX-OS CLI (# guestshell *) Standard Linux docker tool
primitives
Definition of the container Must be done on a Nexus 9K Can be done from any
image content computer supporting Docker
Repository of existing None Docker Hub
container images
Container orchestration None Docker Swarm or Kubernetes
What Apps are Interesting to Host on N9K?
• Monitoring agents
• Open-source agents: OpenTSDB,
Ganglia, Nagios, etc.
Monitor both standard Linux
components (CPU, memory,
interface counters), and NX-OS
(routes, buffers,...)
• Open-source agents: OpenTSDB,
• Custom agents: ECMP load
balancing, PTP accuracy…
• Monitoring agents • Automation agents (Chef, Puppet,

• Open-source agents: OpenTSDB, SaltStack...)

Ganglia, Nagios, etc. • Intrusion Detection

• DNSFlow agent to detect phishing
activity

activity
components (CPU, memory, • Custom Intrusion Detection agents

activity
components (CPU, memory, • Custom Intrusion Detection agents
interface counters), and NX-OS • Automatic configuration backup to
(routes, buffers,...) a private Git repository
Automation
Management Server
Your NX- Native NX-OS Nexus 9K

SDK Apps Features
NX-SDK APIs
NX-OS Infra
(CLI Mgr, Syslog Mgr, ...) CLI
Data Management Engine (DME)

Transaction Commit Object Store Status: Success or Raise Fault
BGP Data VLAN Data QoS Data
Telnet
or
Management Server SSH

SDK Apps Features
NX-SDK APIs
NX-OS Infra

Telnet NX-API
or CLI
SSH Client
Management Server

SDK Apps Features
NX-SDK APIs NGINX Server
NX-OS Infra

Telnet NX-API NX-API
or CLI REST
SSH Client Client
Management Server

SDK Apps Features
NX-SDK APIs NGINX Server
NX-OS Infra

SNMP
or CLI REST
Client
Client
SSH Client
Management Server

SDK Apps Features
SNMP
NX-SDK APIs NGINX Server Agent
NX-OS Infra

SNMP NETCONF / RESTCONF / gRPC
or CLI REST
Client
Client YANG Clients
SSH Client
Management Server

SDK Apps Features

NX-SDK APIs NGINX Server Agent YANG Agents
NX-OS Infra

NX-API
Management
Server
Nexus 9K
{
"jsonrpc": "2.0",
"method": "cli",
CLI "params": {
"cmd": "show version",
Request "version": 1
Management },
Server }
"id": 1
Nexus 9K
{
"jsonrpc": "2.0",
"method": "cli",
CLI "params": {
"cmd": "show version",
Request "version": 1
Management },
Server }
"id": 1
{
"jsonrpc": "2.0",
"result": {
"body": {
"bios_cmpl_time": "03/02/2017",
JSON "bootflash_size": 7906304,
"kickstart_ver_str": "7.0(3)I7(3)",
Response "chassis_id": "Nexus 9508",
... Nexus 9K
Ansible
Writing Your Own Automation Can be
Challenging
Challenging
• NX-API brings a modern transport, and a structured output. But...
Challenging
• Need to have development skills and allocate time to write, maintain and test the
code.
Challenging
code.
• Need to deal with authentication and transport.
Challenging
code.
• Ideally, be able to group switches by category.
Challenging
code.
• Not much leverage of existing, tested apps besides the low-level API.
Challenging
code.
• Not straightforward to re-use existing CLI (Jinja2) templates.
Challenging
code.
• Not idempotent by default, need to implement idempotency yourself.
Challenging
code.
• No parallelization by default, need to implement threading yourself.
Challenging
code.
• No parallelization by default, need to implement threading yourself.
• So let’s leverage existing tools on top of NX-API!
Ansible Overview
Management Server
Ansible Controller
Inventory Modules
Playbooks Config
Configure
Targets
Ansible Overview
• Ansibles automates most DC assets with a

Management Server minimal learning curve.
Ansible Controller
Inventory Modules
Playbooks Config
Configure
Targets
Ansible Overview

• Both switches and servers can be
Ansible Controller managed.
Inventory Modules
Playbooks Config
Configure
Targets
Ansible Overview

• Human-readable → very little scripting
Inventory Modules skills required.
Playbooks Config
Configure
Targets
Ansible Overview

• Agent-less → easy to adopt.
Playbooks Config
Configure
Targets
Ansible Overview

• NX-OS: Ansible modules abstract the CLI.
Playbooks Config
Configure
Targets
Ansible Overview

Playbooks Config • Advanced features:
Configure
Targets
Ansible Overview

Configure
• Variables
Targets
Ansible Overview

Configure
• Variables
• Conditionals
Targets
Ansible Overview

Configure
• Variables
• Conditionals
• Events
Targets
Ansible Overview

Configure
• Variables • Loops
• Conditionals
• Events
Targets
Management NX-API NX-API NETCONF / RESTCONF / gRPC
SSH SNMP
Server CLI REST YANG

SDK Apps Features

NX-OS Infra


Ansible Architecture
Targets
Inventory
SSH
Playbook Config Python
NX-API
NETCONF
Modules
Ansible Architecture • Inventory: target
systems for
automation.
Targets
Inventory
SSH
NX-API
NETCONF
Modules
systems for
automation.
Targets • Playbook: a series
of plays
Inventory (automation tasks).
SSH
NX-API
NETCONF
Modules
systems for
automation.
of plays
• Modules:
accomplish specific
tasks in Ansible
(e.g. install
SSH packages,
Playbook Config Python configure NX-OS,
NX-API etc.)
NETCONF
Modules
systems for
automation.
of plays
• Modules:
accomplish specific
tasks in Ansible
(e.g. install
SSH packages,
Playbook Config Python configure NX-OS,
NX-API etc.)
• Ansible Config:
NETCONF determines how
your Ansible setup
behaves (how many
Modules concurrent
connections, etc.)
Ansible Playbook = Sequence of Tasks to Execute
Example: Deploy NTP on all the Servers
[devops@server ~]$ cat ntp.yml
[devops@server ~]$ cat ntp.yml Playbooks are YAML files

---
- hosts: all
become: yes
tasks:
- name: Ensure NTP is installed
yum: name=ntp state=present
- name: Ensure NTP is running
service: name=ntpd state=started enabled=yes
[devops@server ~]$

---
- hosts: all
become: yes
tasks:
[devops@server ~]$
Blue: Ansible keyword or module

White: a value that you define
Set of target devices (defined in a separate file)

---
- hosts: all
become: yes
tasks:
[devops@server ~]$


---
- hosts: all sudo
become: yes
tasks:
[devops@server ~]$


---
- hosts: all sudo For all those target devices,
become: yes
execute the tasks below
tasks:
[devops@server ~]$


---
become: yes
tasks:
[devops@server ~]$


---
become: yes
tasks:
yum: name=ntp state=present Arguments to
- name: Ensure NTP is running the module
[devops@server ~]$
Ansible module that will do the actual work Blue: Ansible keyword or module
Ansible Inventory
[devops@server ~]$ cat /etc/ansible/hosts

[nxos_vteps]
93180-EX-1
93180-EX-2
92160-1
92160-2
93180-FX-2
[nxos_spines]
9364-1
9364-2
[older_routers]
router-A
router-B
Ansible Inventory
[devops@server ~]$ cat /etc/ansible/hosts [devops@server ~]$ cat \

[nxos_vteps] /etc/ansible/group_vars/nxos_vteps
93180-EX-1 ---
93180-EX-2 ansible_network_os: nxos
92160-1 ansible_connection: network_cli
92160-2 ansible_user: devops
93180-FX-2 ansible_ssh_pass: automate
[nxos_spines]
9364-1
9364-2
[older_routers]
router-A
router-B
Ansible Inventory
[devops@server ~]$ cat /etc/ansible/hosts [devops@server ~]$ cat \

[nxos_vteps] /etc/ansible/group_vars/nxos_vteps
93180-EX-1 ---
93180-EX-2 ansible_network_os: nxos
92160-1 ansible_connection: network_cli
92160-2 ansible_user: devops
93180-FX-2 ansible_ssh_pass: automate
[nxos_spines]
9364-1
9364-2 Best practice: use Ansible Vault for
password encryption.
[older_routers]
router-A
router-B
VXLAN BGP EVPN Automation
[devops@server ~]$ cat vxlan.yml
---
- name: Create L2VNI
hosts: nxos_vteps
tasks:
- name: Create VLAN and map to to VNI
nxos_vlan:
vlan_id: 2200
mapped_vni: 20200
- name: Add L2VNI to Overlay

nxos_vxlan_vtep_vni:
interface: nve1
vni: 20200
...
Blue: Ansible keyword or module name

White: a value that you define TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 249
VXLAN BGP EVPN Automation
[devops@server ~]$ cat vxlan.yml
...
tasks:
- name: Create VLAN and map to to VNI
nxos_vlan: vlan 2200
vlan_id: 2200 vn-segment 20200
mapped_vni: 20200
- name: Add L2VNI to Overlay interface nve1

nxos_vxlan_vtep_vni: no shutdown
interface: nve1 host-reachability protocol bgp
vni: 20200 member vni 20200
multicast_group: 239.239.239.100 suppress-arp
suppress_arp: true mcast-group 239.239.239.100
What’s the Cleanest Way to Handle This Error?
[devops@server ~]$ cat vxlan.yml Config prior to the playbook
... execution:
tasks:
- name: Create VLAN and map to to VNI interface nve1
nxos_vlan: no shutdown
vlan_id: 2200 host-reachability protocol bgp
mapped_vni: 20200 member vni 20200
admin_state: up suppress-arp
ingress-replication protocol bgp
interface: nve1
vni: 20200
multicast_group: 239.239.239.100
suppress_arp: true
Checkpoint and
Rollback!
[devops@server ~]$ cat rollback.yml
Checkpoint and ...
tasks:
Rollback! - name: Create checkpoint
nxos_rollback:
checkpoint_file: backup.cfg
Checkpoint and ...
tasks:
nxos_rollback:
- name: VXLAN config

block:
- name: Create VLAN and map to the VNI
nxos_vlan:
vlan_id: 2200
mapped_vni: 20200
admin_state: up
interface: nve1
vni: 20200
Checkpoint and ...
tasks:
nxos_rollback:
- name: VXLAN config

block:
- name: Create VLAN and map to the VNI
nxos_vlan:
vlan_id: 2200
mapped_vni: 20200
admin_state: up
interface: nve1
vni: 20200
rescue:
- name: Rollback to checkpoint
nxos_rollback:
rollback_to: backup.cfg
[devops@server ~]$ ansible-playbook rollback.yml
TASK [Create checkpoint]
*************************************************************************************************
changed: [93180-FX-2]
TASK [Create VLAN and map to the VNI]

*************************************************************************************************
TECDCN-2002 TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 253
*************************************************************************************************

*************************************************************************************************
TASK [Add L2VNI to Overlay]

*************************************************************************************************
fatal: [93180-FX-2]: FAILED! => {"changed": false, "clierror": "Cannot associate a multicast group
or vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error", "output":
[{"body": {}, "code": "200", "msg": "Success"}, {"body": {}, "code": "200", "msg": "Success"},
{"body": {}, "code": "200", "msg": "Success"}, {"clierror": "Cannot associate a multicast group or
vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error"}], "url":
"http://93180-FX-2:80/ins"}
TECDCN-2002 TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 254
*************************************************************************************************

*************************************************************************************************
TASK [Add L2VNI to Overlay]

*************************************************************************************************
fatal: [93180-FX-2]: FAILED! => {"changed": false, "clierror": "Cannot associate a multicast group
or vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error", "output":
[{"body": {}, "code": "200", "msg": "Success"}, {"body": {}, "code": "200", "msg": "Success"},
{"body": {}, "code": "200", "msg": "Success"}, {"clierror": "Cannot associate a multicast group or
vrf to an ingress replication vni.\n\n", "code": "400", "msg": "CLI execution error"}], "url":
"http://93180-FX-2:80/ins"}
TASK [Rollback to checkpoint]
*************************************************************************************************
PLAY RECAP
*************************************************************************************************
93180-FX-2 : ok=5 changed=3 unreachable=0 failed=1
[devops@server ~]$ TECDCN-2002 TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 255
NX-OS Ansible Modules
Over 70 NX-OS Modules in Ansible 2.7
• AAA
• ACL
• BGP
• Checkpoint / Rollback
• CLI
Note: can be used with
Jinja2 templates
• HSRP / VRRP
• IGMP / IGMP Snooping
• AAA • Interfaces
• ACL • NTP
• BGP • NX-API
• Checkpoint / Rollback • NX-OS Facts
• CLI • OSPF
• PIM
Jinja2 templates
• HSRP / VRRP • Port-Channel / vPC
• Patching
• AAA • Interfaces • Snapshot
• ACL • NTP • Static Routing
• BGP • NX-API • Upgrade
• Checkpoint / Rollback • NX-OS Facts • VLAN
• CLI • OSPF • vPC

• PIM • VRF
Jinja2 templates
• HSRP / VRRP • Port-Channel / vPC • VXLAN Flood & Learn
• Patching • VXLAN BGP EVPN

• AAA • Interfaces • Snapshot
• ACL • NTP • Static Routing
• BGP • NX-API • Upgrade
• Checkpoint / Rollback • NX-OS Facts • VLAN
• CLI • OSPF • vPC

• PIM • VRF
Jinja2 templates
• HSRP / VRRP • Port-Channel / vPC • VXLAN Flood & Learn
• Patching • VXLAN BGP EVPN

Most extensive support for networking software in the industry.

NETCONF/YANG
Read all VRFs with OpenConfig
<get-config>
<source>
<running/>
</source>
<filter>
<network-instances xmlns="http://openconfig.net/yang/network-instance">
<network-instance/>
</network-instances>
</filter>
</get-config>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2">
<data>
<network-instances xmlns="http://openconfig.net/yang/network-instance">
<network-instance>
<config>
<description/>
<enabled>true</enabled>
<name>default</name>
<type>L3VRF</type>
</config>
<name>default</name>
</network-instance>
<network-instance>
<config>
<description/>
<enabled>true</enabled>
<name>Testing1</name>
<type>L3VRF</type>
</config>
... TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 260
SNMP
or CLI REST
Client
Client YDK
SSH Client
Management Server

SDK Apps Features

NX-OS Infra

Prototyping and
Validating with
N9Kv
Nexus 9000v (N9Kv)
Nexus 9000v (N9Kv)
• N9Kv is:
Nexus 9000v (N9Kv)
• N9Kv is:
• A simulation platform that runs N9K NX-OS as a virtual machine.
Nexus 9000v (N9Kv)
• N9Kv is:
• Abstracting N9K hardware: supervisor, one line card, interfaces.
Nexus 9000v (N9Kv)
• N9Kv is:
• Using a software data-plane.
Nexus 9000v (N9Kv)
• N9Kv is:
• Supported in VIRL.
Nexus 9000v (N9Kv)
• N9Kv is:
• N9Kv is not:
Nexus 9000v (N9Kv)
• N9Kv is:
• N9Kv is not:
• A replacement for N9K physical platform.
Nexus 9000v (N9Kv)
• N9Kv is:
• N9Kv is not:
• Designed to be used as a switch in a production data center.
Nexus 9000v (N9Kv)
• N9Kv is:
• N9Kv is not:
• Designed to be used as a switch in a production data center.
• Supported hypervisors: ESX, KVM, VirtualBox, Fusion.

NX-OSv Is Great for Prototyping and Testing
• DevOps
• DevOps
• Rapid prototyping of 3rd party tools & applications
• DevOps
• Rapid development & testing of platform-independent features
• DevOps
• Configuration policy impact analysis
• DevOps
• Validate network configuration prior to deployment in the actual network
• DevOps
• Analyze control plane behavior
• DevOps
• Lab as a Service
• DevOps
• No physical test beds required
• DevOps
• No physical test beds required
• Learning Tool
Matthias Wessendorf
Data Center Technical Marketing Engineer
Network Manager
(DCNM)
DCNM Overview &
Functions
Data Center Network Manager (DCNM)
Single Management Solution for
• VXLAN EVPN Programmable

Fabric
• Classic LAN Deployments
• IP Media Network Controller
(PMN)
• SAN (MDS & Nexus)
• VXLAN EVPN Programmable

Fabric
• Classic LAN Deployments
(PMN) Addresses End-to-End
• SAN (MDS & Nexus) Network Provisioning
• GUI/API-based
provisioning
• Multi-Fabric & Multi-
Site
• Network Configuration
Backup & Restore
Helps in Day 2 Operations

• Real-Time Topology
• Integrated Compute Visibility
• Performance Monitoring
• VXLAN EVPN Programmable • Fault Management
Fabric • Configuration Compliance
• Classic LAN Deployments • Image Management, Upgrades
and RMA
(PMN) Addresses End-to-End
• SAN (MDS & Nexus) Network Provisioning
• GUI/API-based
provisioning
• Multi-Fabric & Multi-
Site
• Network Configuration
Backup & Restore
Network Architecture Deployment Models ~
DCNM Modes
DCNM Modes
Fabric / Overlay Models
VXLAN + BGP-EVPN
• L2 over L3 overlay
• BGP-EVPN control plane
• VXLAN data plane
DCNM Modes
Fabric / Overlay Models Traditional Models
VXLAN + BGP-EVPN Traditional L2 / L3
• L2 over L3 overlay • L3 @ aggregation & L2 @ access
• BGP-EVPN control plane • L3 @ access

• 3-tier or spine-leaf model
DCNM Modes

LAN Fabric Mode – Easy Fabric Template

(Nexus 3k/9k)
DCNM Modes

LAN Fabric mode - External Fabric Template

LAN Fabric Mode – Easy Fabric Template
LAN Classic Mode
(Nexus 3k/9k)
(Nexus 2k-9k)
Introducing LAN fabric in DCNM 11 for Nexus
3k/9k
4
Deploy
Centralized config push Define
Define Intent based on best practices
1 • Underlay
• Interfaces
• Overlay
3
Preview
Side-by-side diff Save
Generates configuration based on intent
2
Getting your fabric up and running
DCNM for New or
Existing Fabrics
Classic LAN and External Fabric - Nexus 2k-9k
Use within LAN Fabric mode Separate Install Mode
With configuration compliance No configuration compliance
Backup and restore of configs Backup and restore of configs

and state
External Classic
Flexible Python + templates Fabric LAN CLI based templates
Endpoint locator No Endpoint locator
One-click vPC pairing vPC wizard
Bootstrap/VDC POAP*/SMU patching Switch and VDC POAP/SMU patching
* roadmap TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 271
Leveraging DCNM for New and Existing Fabrics
Install / Use DCNM

(Virtual Appliance)
Bootstrap Devices
Discover Existing
[POAP]
Fabrics / Networks
STP/VPC
Fabric Builder
Create New VXLAN
DFA / FabricPath
Fabrics
Maintain and Operate New VXLAN Fabrics

& Existing VXLAN or FabricPath or VLAN Fabrics
VXLAN Fabric Builder Auto-Deployment
Managed-
Turn-Key VTEP Simplified
Fabric
Deployment Deployment
Operations
VXLAN Fabric Builder Auto-Deployment
VXLAN Fabric
Auto-Deployment
Managed-
Turn-Key VTEP Simplified
Fabric
Deployment Deployment
Operations
Day 1+ Operations
Manage, Monitor, Visualize, Search
Challenge: Manage & Grow Underlay with minimal overhead & keep consistent intent
Day 1+ Operations
Manage, Monitor, Visualize, Search
Challenge: Manage & Grow Underlay with minimal overhead & keep consistent intent
Deployed Fabric Manage Monitor / Visualize /
Search / Update
Underlay:
• SDN Networks [VTEPs]
• Image Update [ISSU]
• View Fabric Topology
• Monitor Health, Events,
Performance
[cpu/mem/iface/syslog]
• Add Devices/Expand
Cisco Advantage:
• Turnkey Management
• Integrated Views
• Comprehensive Fabric Views
Day 1+ Operations
Overlay Visibility, Growth
Challenge: Manage / Monitor SDN Overlay’s across a large fabric
Overlay Tasks: Monitor / Visualize / Search
• Visualize Overlays [VXLAN, VLAN, etc..]
• Add, Manage SDN Networks
• Shows VM Networking Path [vCenter]
• Find, Track VMs, Workloads [EPL]
• Find VN’s and VNI’s [VXLAN]
• View VXLAN E2E Connectivity [OAM]
• Identify Errors
• Validate Compliance
Cisco Advantage:
• Seamless Overlay/Underlay Correlation
• Easy to find workloads, VN’s, VNI’s on vast
fabric
• Easy to See Host-Network chain
Day 1+ Operations
Overlay Visibility, Growth
Challenge: Manage / Monitor SDN Overlay’s across a large fabric
Overlay Tasks: Monitor / Visualize / Search
• Visualize Overlays [VXLAN, VLAN, etc..]
• Add, Manage SDN Networks
• Shows VM Networking Path [vCenter]
• Find, Track VMs, Workloads [EPL]
• Find VN’s and VNI’s [VXLAN]
• View VXLAN E2E Connectivity [OAM]
• Identify Errors
• Validate Compliance
Cisco Advantage:
• Seamless Overlay/Underlay Correlation
• Easy to find workloads, VN’s, VNI’s on vast
fabric
• Easy to See Host-Network chain
Day 1+ Operations
Verify Compliance
Challenge: Ensure Deployment [Underlay, Overlay, Access] is Correct
Compliance Tasks: Detect and Fix
• Monitor Fabric
• Compare device configuration
against Fabric policy
• Remediate [revert or change Policy]
On-Demand
remediation
Cisco Advantage:
• Constant Monitoring
• Compliance engine brings fabric back to
intended configuration
• No un-anticipated behavior
Compliance engine remediates to intended configuration
Getting Started
with DCNM
VXLAN User Experience with DCNM
• DCNM Differentiates Underlay vs. Overlay

• Use Fabric Builder

OR

OR
• Use POAP templates for Underlay configuration
• VXLAN Best-Practice Templates from cisco.com
• Basic Manageability for “Classic” configurations

OR
• Deploy Network/VRFs using “LAN Fabric Provisioning” for Overlay

OR
• Deploy Network/VRFs using “LAN Fabric Provisioning” for Overlay

• View VXLAN details via Topology Views, Search & Multi-Site
Discovering the Data Center
• DCNM Data Sources Include: SAN, LAN, VMware, & Storage Arrays
POAP will automatically start discovery, so you won’t need to do this if

you bootstrap via Fabric Builder or Classic POAP
VXLAN Underlay Bring-Up – DCNM Starting Point
Use Virtual
Appliance (VA)
-VA Includes Fabric
Infrastructure
Configure DCNM
Use Virtual Fabric
Appliance (VA) Management
(OVA / ISO Setup)
-VA Includes Fabric
Infrastructure -Management IP
-Fabric Management
subnet
Configure DCNM
Use Virtual Use Fabric Builder
Fabric
[regular POAP for
(OVA / ISO Setup)
‘classic’ mode]
-VA Includes Fabric
--Generate POAP
definitions
-Fabric Management
subnet
Configure DCNM
Fabric
[regular POAP for
(OVA / ISO Setup)
‘classic’ mode]
-VA Includes Fabric
--Generate POAP
definitions
-Fabric Management
subnet
Deploy Fabric
-Switch VTEP
Configures
Automatically during
POAP
Configure DCNM
Fabric
[regular POAP for
(OVA / ISO Setup)
‘classic’ mode]
-VA Includes Fabric
--Generate POAP
definitions
-Fabric Management
subnet
Deploy Fabric
-Switch VTEP
Fabric
Configures
Automatically during
Underlay
POAP Installed
VXLAN Fabric Builder…
NEW in
DCNM 11
1 Pre-Stage, Minimal Input 3 Discover & Bootstrap
2 Assign Role & Deploy 4 Inspect New Fabric
Auto-VPC
VXLAN Fabric Compliance
1 3 Deploy Changes
Out-of-Sync
Detected
2 Preview Compliance
Remediation
4 Fabric Repaired!
LAN Fabric Brownfield Deployment
• Non-disruptive import of existing VXLAN EVPN deployments

• Learns topology, all configuration, associated resources, IP subnets, VNIs, VLANs etc. from the
existing deployment
• State reverse populated into DCNM
• Start managing fabric as if provisioned from DCNM

Information Store
Underlay Interfaces Overlay
Inference
config config config

config
Brownfield Migration
VXLAN-EVPN Fabric Builder
Create Fabric
• Procedure:
• Fabric Definition & Creation
Define Fabric
• Import Devices via Add Switches
• Switches will be imported in “Migration” State
• Save & Deploy Import Switches
• Per switch configuration processing matched against selected Fabric Settings
• All configuration & associated resource usage per switch learnt by the DCNM
Assign Switch Role and
• Sanity checks for mis-configs with error reporting Create VPC Pairing (Optional)
• Iterative process to continue migration post error correction
Config Review and Deploy
Let’s Focus on
New VXLAN
Functions
Top Down Deployment
• Deploys Configuration Profile for VRF,

Segment or Interface to Switch -VRF
without a Trigger -VNI
• Pushed from DCNM GUI or REST API -VLAN

-(Interface)
• Doesn’t Require Switch Auto-
configuration
• Select an existing fabric or add a new
fabric and then define Fabric Settings.
Support the VXLAN-EVPN for N9K.
Adding A New VXLAN Fabric
1. Create a New VXLAN Fabric ‘on the fly’

2. Use or adjust default settings
3. Update Pools for this fabric as necessary
Creating A New Network
1) Use Default VNI or

adjust
1) Choose VRF or add new
2) Add G/W to Define Net
Deploying The Network
1) Select Network
Staged Deployment is Blue

Yellow is “Deploying” 2 ) Choose which Switches to Deploy to
Green indicates Success
Red indicates failure
3) Deploy
Deploying The Network – Selecting Switches
Double-Click Deploy to this switch Ready to Deploy

[De-Select to remove Network]
Select Ports if desired

(Not necessary if default is ‘trunk’)
In Progress Deployed
• Double Click the Switches where you want the network

• Select “Apply to Switch”
• Select Deploy
• Green indicates success
Controls
Deploy Details
Show / Troubleshoot Deployment
Preview
Add Switches to
Fabric
Refresh
Auto-Refresh
on/off
External Fabric Connectivity Provisioning
Border Node Deployments

• Setting up base and setup
configuration
• Deploying VRFs
• Deploying VRF_LITE using sub-

interfaces with pool management
of dot1q IDs
• IPv4 & IPv6 support
• VPC Support
• Deploying Networks for vanilla

VLAN hand-off
VXLAN Multi-Site Deployment
Fabric 1 Fabric 2
VNI 34112
VNI 26214
Multi-Fabric Simplified/Coordinated Managed-Fabric

Management Operation Operations
Fabric 1 Fabric 2
Border Leaf
B Extensions B
VNI 34112
VNI 26214

Fabric 1 Fabric 2
Border Leaf
B Extensions B
VNI 34112 VNI 34112
VNI 26214 VNI 26214

EVPN Multi-Site Deployment
Support for Border Gateways

• Multi-Site Underlay & Overlay
inter-fabric connection setup
• B2B and Route-Server based
topology support
• Multi-Site Overlay extension

• Networks & VRFs
• Simultaneous VRF-LITE & Multi-

Site support
Troubleshoot VXLAN Using OAM
Show Fabric
Reachability
Helps Troubleshoot
Problems
Switch to Switch or
Host to Host
L4-L7 Service Attachment Use-cases
Virtual & Physical Form Factor

Static & Dynamic Peering
vPC/Non-vPC Attachments
Intra-tenant/Inter-tenant One-armed/Two-armed
• PBR Use-cases • PBR Use-case (No SNAT)

• Tenant-Edge Firewall
L4-L7 Service Attachment GUI
• The anchor screen lists the defined service nodes and associated route peering and service
policies for a selected easy fabric.
• Enabling/disabling the route peering and service policy will cause the corresponding network
and VRF configuration to be updated. User can preview the generated configurations on
involved switches and deploy them on one shot. Select
Fabric
• User can export/import route peering and service policies.
enable/attach
service policy
Topology – Redirected Flow
• The redirected flow section is added to the switch info overlay screen when user double-clicks the icon of the
switch, which has service configured network attached, on the topology.
Show more flows
VXLAN / Fabric Recap...
• Multiple Fabrics on the same pane of glass

• Best Practice Templates for Underlay Provisioning
• Easy Overlay Deployment incl. L4-L7 Services
• Manage Classic Configurations and new Fabrics
• VPC / STP networks
• Fabricpath
• VXLAN
• View VXLAN and Fabricpath on the Topology [Search Details]

• VXLAN-OAM shows fabric reachability
• Endpoint Locator tracks VM lifecycle – “Where’s my VM”?
Cool Features for
LAN Fabrics
(Let’s Explore)
Features in DCNM
Top-Down Provisioning
Topology Views
Real-Time
Search
Detected VTEP
Health Score
(color)
Link Pop-Up Pop-Up Switch
Dashboard
Topology Views- VMM Integration
Display connected
Physical Hosts
Display DVS/Vswitch
Display VMs
Filter by VMM
Display Host Details
Display connected
Physical Hosts
Display DVS/Vswitch
Display VMs
Filter by VMM
Display Connectivity
Details
Display connected
Physical Hosts
Display DVS/Vswitch
Display VMs
Filter by VMM
Display Connectivity
Details
Display connected
Physical Hosts
Display Port-Group
Display DVS/Vswitch
Details
Display VMs
Filter by VMM
Topology Views- Kubernetes Integration
(Preview)
Container option
added
Pod List
available
Endpoint Locator (EPL)
Number of Endpoints Number of Active VRFs
New – Support for Dual

Stacked Endpoints
Number of Active
Network New – Recent
Notifications
Switch Dashboard Interfaces
Interface Page Programmable Show Commands
Interface History
Controls Configuration by Policy [Micro Template]
Policy
Add I/F Edit shut / no shut Show
History
Template Library
VXLAN Policy Micro-

POAP General CLI Show [cli]
Profile Templates
Resource Manager
• Day 2 Operations - Resource ManagementL2/L3 VNI
Networks and
VRFs
Resource Manager
Resource Manager
MSD FABRIC
• Day 2 Operations - Resource ManagementL2/L3 VNI
Devices
Networks and
VRFs
Deployment
Type
Serial
Number
Resource Manager
Built-In REST API-Docs Using Swagger
https://<dcnm-server-IP>/api-docs
DCNM GUI uses
REST API
Inspect with
Browser Tools
[e.g. . Google
Developer Tools]
Automate
DCNM REST APIs for Automation are built-in

Exploring .. Summary Dashboards
Add & remove Dashlets

on demand
Customize for your

environment
Azeem Suleman
Principal Engineer
Application Centric
Infrastructure (ACI)
ACI: An Innovative Approach to Policy Based
Segmentation
Segmentation
APPLICATION
NETWORK PROFILE
Segmentation
APPLICATION
NETWORK PROFILE
Segmentation
INTERNET
WEB APP DB
APPLICATION
NETWORK PROFILE
Segmentation
INTERNET
F/W WEB ADC APP DB
ADC
APPLICATION
NETWORK PROFILE
What is an application network policy?

1. Group: A set of workloads spread across clouds with the same policy
2. Contracts: A set of rules governing communication between groups
3. Service Chains: A set of network services between groups
Application Policy Infrastructure Controller (APIC)
Root
Policy Virtual
Infra Fabric
Universe Network
Tenants VLANs Nodes Hypervisors
Applications
Root
Policy Virtual
Infra Fabric
Universe Network
Applications
View and Manage the Fabric as a System
Root
Policy Virtual
Infra Fabric
Universe Network
Applications

•Enables Shared Visibility
Root
Policy Virtual
Infra Fabric
Universe Network
Applications

Data Base Enabled and Defined Networking by Logical Views
Root
Policy Virtual
Infra Fabric
Universe Network
Applications

Data Base Enabled and Defined Networking by Logical Views
•Enables Shared Context to Simplify Operations
ACI: The Elements
APICs L-Size (Recommended VMware VMs

3 Recommended for Production for 1000+ physical leaf ports) (Recommended for 2-4 leaves, Cloud
Physical Virtual 2 VMs + 1 Physical APIC)
At least 1 physical APIC required M-size (Recommended
for <1000 physical leaf ports)
ACI: The Elements

SPINES Modular Fixed Nexus

9300
Nexus 9500 (9332C, 9364C)
(w/9700 LCs)
ACI: The Elements

Virtual
SPINES Modular Fixed Nexus vPod
9300 (vSpine)
Nexus 9500 (9332C, 9364C)
(w/9700 LCs)
LEAVES Virtual
Virtual/Container networking integration Fixed Nexus 9300 vPod
included (except vPod mode) (100M/1/10/25/40/50/100/400G) (vLeaf)
ACI: The Elements

Virtual
SPINES Modular Fixed Nexus vPod
9300 (vSpine)
Nexus 9500 (9332C, 9364C)
(w/9700 LCs)
LEAVES Virtual
Virtual/Container networking integration Fixed Nexus 9300 vPod
included (except vPod mode) (100M/1/10/25/40/50/100/400G) (vLeaf)
Premier Insights &

Assurance Add-ons
vPod
LICENSING Advantage Multisite
Remote-Leaf Mgmt Cluster +
Per AVE License
FC
FCoE
Storage Encryption
Essentials
The DC network before
Classic modular switching
Supervisors (1 or 2)
Up to 18 RUs Scale-up
Single chassis (e.g. Nexus 7000)
Linecards (Copper, Fiber,1/10G)
Fabric Modules (3- 5)

The DC network before The DC network NOW
Classic modular switching ACI
APICs
(1,3 or more)
SPINE
(2 to 6)
Zero-touch VXLAN
No STP
LEAVES
(1 to 200 or more*)
Single VXLAN Network**
Evolution from Nexus 5000 and Nexus 7000
The DC network before The DC network NOW
Classic modular switching ACI
APICs
(1,3 or more)
SPINE
(2 to 6)
Zero-touch VXLAN
No STP
LEAVES
(1 to 200 or more*)
Scale as you need
Single VXLAN Network**
Evolution from Nexus 5000 and Nexus 7000
Single chassis (e.g. Nexus 7000) * > 200 Leaves with MultiPod/Multi-Site
** Other topologies available (e.g. 3-tier, etc.)
Cisco, as Open as
You Want it to Be
Be Open Automation Ready
• Expose Network Services

• From build to ongoing support
• Policy based management
• Open API’s, SDK, object models
ACI: How difficult is it to bring it up?
Let’s start with a single site
Spine Layer Nexus 9000
Leaf Layer Nexus 9000

1 Connect all leaves to spines
Connect APIC(s) to any leaf or leaves

2 Console into to each of the APICs

Follow the initial configuration wizard
LLDP
Exchange
• ISIS protocol is run on links

TEP through
DHCP between spines / leaves
ISIS Protocol
Adjacency
Certificate
2
Validation
Console into to each of the APICs
DME Start
LLDP
Exchange
• ISIS protocol is run on links

TEP through
DHCP between spines / leaves
ISIS Protocol
Adjacency
Certificate
2
Validation
Console into to each of the APICs
DME Start
• Spine – OOB, Inband management and 1 console per Sup for 95xx
• Leaf – OOB, Inband management and 1 console
• APIC – CIMC and dual home connection, standby APIC (if possible)
• Fabric Name, Fabric ID, Infra TEP Pool /22, Infra VLAN(3967), BD Multicast Range, NTP, AAA
• Export backups / snapshots periodically
What tasks & configuration did ACI just saved me from doing manually on every switch
BEFORE NOW
SSH to every switch, Assign IP Address, Enable

Telnet/SSH, Add users on every switch/Create ACLs
(optional)
(Times X Switches & Y VNIs)
What tasks & configuration did ACI just saved me from doing manually on every switch
BEFORE NOW
External to Internal Route
redistribution (MBGP)
Multicast and Control Plane
(MBGP)
Overlay Network (VXLAN)
Underlay Routed Network (IS-IS)
Switch management (Inband or

Out-of-Band options)
SSH to every switch, Assign IP Address, Enable
Telnet/SSH, Add users on every switch/Create ACLs ACI Automated tasks
(optional)
From HOURS to seconds!
(Times X Switches & Y VNIs)
Key Concepts &
Recommendations
What is Tenant
Tenant A Tenant B
A Tenant is a container for all
network, security,
troubleshooting and L4 – 7
service policies.
Tenant resources are isolated

from each other, allowing
management by different
administrators.
Where VRF is defined
Tenant A Tenant B
VRF 1 VRF 1 VRFs (contexts) are defined
within a tenant to allow
isolated and potentially
overlapping IP address
space.
VRF 2 VRF 2
What is Bridge Domain (BD)
Tenant A Tenant B Within a VRF (Context), one

VRF 1 VRF 1
or more bridge domains must
be defined.
Bridge Domain 1 Bridge Domain 1
Bridge Domain 2 Bridge Domain 2 A bridge domain is a L2

forwarding entity within the
VRF 2 VRF 2 fabric, used to define L2
forwarding domain and to
constrain broadcast and
multicast traffic.
Bridge Domain
Forwarding L2 unknown unicast based on spine-proxy mapping database
or flood and learn over VXLAN
THIS DOESN’T CONTROL THE BEHAVIOR OF L3 traffic, i.e. it doesn’t

control the forwarding of L3 "unknown" unicast
THIS doesn’t turn on or off the mapping database for MAC addresses.
MAC addresses are always learned in the mapping database
L3 Multicast (IANA Range). Known multicast traffic will have IGMP/MLD

snooping entry and forwarded to appropriate ports
Unknown multicast will get FLOOD to ports in BD or
In optimize flood case send only to router ports detected by PIM hellos
This option is only relevant if you do hardware-proxy forwarding and if “Unicast routing” is
enabled. ARP packets are flooded in the BD
Learn only IP addresses from configured subnets – It’s disruptive
Bridge Domain Recommendations
Enforce
Subnet
L2 Unknown ARP Unicast Subnet
Scenario Check for
Unicast Flooding Routing Configured
IP
Learning
IP Routed Traffic. No FW + LB, No Floating IP Hardware Proxy Disabled Enabled Yes (if Yes
required. No Silent Hosts required)
IP Routed Traffic. No FW + LB. Silent Hosts. Hardware Proxy Disabled Enabled Yes Yes
Non IP, switched traffic. Silent Hosts. Flood N/A Disabled No N/A
Hosts with IP address may float between Hardware Proxy Enabled Enabled Yes Yes
MAC. FW + LB. NIC Teaming
Migration – Extending L2 from ACI with L3 Hardware Proxy Enabled Enabled If required If required
GW still on legacy network
• Start with HW-Proxy ON unless if deemed necessary

• ARP Flooding on for L4-7 devices failover process
• Data plane learning on (except for Service Redirect)
L2 Forwarding Reaches
Remote leaf
that floods it
Forward to
remote Leaf
Yes Does spine knows
Dst Mac
No
Drop
Summary
Forward to Forward to Flood within
local port remote leaf BD Spine Proxy
Flood Hardware Proxy
Is Dst MAC on What is BD config?

Local Leaf? (L2 Unknown Uni)
Yes No
Does Leaf know

Dst MAC?
Yes No See later pres
L2 or L3 ?
L2
Packet coming in to Leaf

ACI: What changes?
Easy as 1-2-3-4-5
BEFORE
Physical Networks/VRFs
We would purchase separate networks and assign
different IP subnets to each (Prod, Test, etc.)
Test Production
2.2.2.0 IP Change 1.1.1.0
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
1
Physical Networks/VRFs Create Tenants
We would purchase separate networks and assign You can ”partition” your ACI Fabric & have up to 3000 Tenants even
different IP subnets to each (Prod, Test, etc.) using the same IP subnets with no conflict
Tenant Test Tenant Prod

1.1.1.0 1.1.1.0
Test Production
2.2.2.0 IP Change 1.1.1.0
What is End Point Group (EPG)
Tenant A Tenant B
EPGs exist within a single
VRF 1 VRF 1 bridge domain only – they do
Bridge Domain 1 EPG Bridge Domain 1 EPG not span bridge domains.
Bridge Domain 2 EPG Bridge Domain 2 EPG
EPGs defines the policy
enforcement entities/classes.
VRF 2 VRF 2
Class-based policies are
Bridge Domain 3 EPG EPG Bridge Domain 3 EPG EPG
applied between EPGs
Bridge Domain 4 EPG Bridge Domain 4 EPG
Application Policy Logical Construct
Tenant
VRF 1 VRF 2
Network
Bridge Domain 172 Bridge Domain 10 Bridge Domain 100
Subnet 172.1.1.0/24
Subnet 10.1.1.0/24 Subnet 10.1.1.0/24
Subnet 172.1.2.0/24 Subnet 10.1.2.0/24
… …
Subnet 172.20.1.0/24
EPG web
EPG WEB EPG DB Policy “HTTP”
App
Policy “HTTP”
EPG db
Policy “SQL”
EPG APP Policy “SQL”
EPG app
Application Policy Logical Construct
Mapping the Configuration to the Packet
Coke-Tenant
• ACI Fabric leverages VXLAN Encapsulation to build VRF 1
network overlay
Bridge Domain 1 EPG
• VXLAN Source Group is used as a tag/label to identify the
specific end point for each application function (EPG) Bridge Domain 2 EPG
• Policy is enforced between an ingress or source application

tier (EPG) and an egress or destination application tier
(EPG) VRF 2
• Policy can be enforced at source or destination Bridge Domain 3 EPG
Bridge Domain 4 EPG
VXLAN Header:
Flags Flags/DRE Source Class ID == EPG VNID == BD/VRF M/LB/SP
End Point Group (EPG) Definition
• An Endpoint group (EPG) is a set of devices (end points) that share the same policy
requirements.
• Classification can be based on:
• VLAN Application
Profile
• VxLAN
• MAC Address
• IP Address EPG EPG
• VM Properties etc.
EP EP EP EP
Virtual Port, Physical Ports, External L2 VLAN, External L3 subnet
End Points (EPs)
• EPs are devices which attach to the network either virtually or physically, e.g.:
• Virtual Machine
• Physical Server (running Bare Metal or Hypervisor)
• External Layer 2 / 3 device
• Firewall / Load balancer etc.
ACI Endpoint
Traditional Endpoint
- MAC or MAC/IP → IPv4 is /32
L2 – MAC Table L3 – ARP Table Route
- MAC Address - IP / MAC - VLAN / VxLAN → EPG (pcTag)
- VLAN - Interface
- Interface - Interface - VRF
- VRF - Flags → Local, vPC, static, etc.
Endpoint Classification
Web App DB
Outside QoS QoS QoS

(Tenant
VRF) Filter Service Filter
Classification:
Classification: Classification: Classification:
L3_Out : Network/Mask
• EPG Classification on L3 Outside • EPG Classification on an access/server port is based on different attributes
is based on IP address • Port + VLAN, Port + VXLAN, Network/Mask
Network/Mask
• IP/MAC, VM Attributes for AVS attached VM’s
• IP & MAC Host Address
Understanding Networks and Groups
Abstractions
Legend
Tenant Child/Parent Object
(fvTenant) Relationship (Pointer)
Application Bridge
Outside VRF Contract
Profile Domain Filter
Network (fvAp) (fvBD)
(fvCtx) (vzBrCP)
Subnet
(fvSubnet)
Subject
Endpoint
Group
(fvAEPg)
By default …
endpoints in different EPGs can NOT

communicate at all
By default …
endpoints inside an EPG can

communicate freely
Intra EPG default can be changed
… today, to block intra-EPG communication
Every EPG belongs to a VRF and
an Application Network Profile
Application Network Profile
A group of EPGs related to each other to represent an application
EPG, uEPG, domain associations,

contract relations and L4-7
Configuration
Health scores, statistics, logs

and audit data automatically
correlated and rolled up at
Application Profile level
EPG, uEPG, domain associations,

contract relations and L4-7
Configuration
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE
Unclear network connectivity
Show VLAN would show all and every VLAN per-Switch
without understanding how they connect between each
other
Switch 1 Switch 5
Switch 2 Switch 6
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
Unclear network connectivity 2
Create Application Profiles
Show VLAN would show all and every VLAN per-Switch An Application Profile is a graphical representation of our network
without understanding how they connect between each configuration. Think of it as a “folder of VLANs” at the Fabric level.
other A Tenant may have multiple Application Profiles
Application Profile Learning App
Switch 1 Switch 5
Switch 2 Switch 6
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE
Create VLANs per Switch
Add VLANs per Switch, name each of them and then configure
trunks to extend connectivity. Additionally configure HSRP/VRRP
for Gateways at the distribution/core layer
Collapsed HSRP/VRRP
Core
Gateways
802.1q
Access Layer
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
Create VLANs per Switch 3
Create End Point Groups (EPGs)
Add VLANs per Switch, name each of them and then configure We will create an EPG and name it just as we would with a VLAN. You
trunks to extend connectivity. Additionally configure HSRP/VRRP may also add one Bridge Domain per EPG with an IP address (just like
for Gateways at the distribution/core layer an SVI) in case you want ACI Anycast Gateway functionality
Collapsed HSRP/VRRP Spine Layer

Core Tenant Production
Gateways Application Profile SAP
VXLAN
802.1q
Anycast GW Leaf Layer
Access Layer
EPG EPG
Netweaver HANA
Contracts
• Contracts are semantics to specify EPG to EPG communication in ACI
• Communication policy includes filters (ACLs), QoS and Service Graphs
• Contract filters are similar to Access Control Lists
• Contracts can be defined between EPGs or between L3out External EPGs and regular
EPGs
Contract - MyContract
Subject
Web-Prod Filters DB-Prod
QoS
Service
Graph
EPGs Provide and/or Consume Contracts
• EPGs will have associations to provide

and/or to consume a contract
• An EPG can provide and/or consume
multiple contracts.
• Contracts can be used between EPGs in the
same Application Profile, across Application
Profiles, VRFs and even tenants.
• Contracts also define route-leaking
between VRFs
EPGs Provide and/or Consume Contracts
• EPGs will have associations to provide

and/or to consume a contract
• An EPG can provide and/or consume
multiple contracts.
• Contracts can be used between EPGs in the
same Application Profile, across Application A contract between VRFs is
required to enable route-leaking.
Profiles, VRFs and even tenants.
• Contracts also define route-leaking
between VRFs
Contract Scope Defines Where They Will Be
Applied
• Contract “scope” limits the type of relations between
EPGs.
• Application Profile:
• Contract is applied between EPGs if they are of the
same AP
• VRF:
• Contract applied between EPGs if they are part of the
same VRF.
• Tenant:
• Contract applied if EPGs are in the same tenant, even
if different VRF
• Global:
• Contract can be exported, and is applied even if EPGs
may be part of different tenants
Contract Filters Define L2-4 Traffic
Filters can be re-used by many contracts
Filter definition allows all parameters for L2-4 filtering
A filter may have multiple entries, each matching

specific protocols, ports or port ranges.
Filter definition allows all parameters for L2-4 filtering
Contract Filters Define L2-4 Traffic (contd.)
The Established flag is designed to allow

TCP traffic of existing connections: ACK or
RST
The Established flag is designed to allow

TCP traffic of existing connections: ACK or
RST
The stateful option can be used with AVE in order to

create a reflexive ACL to allow a specific TCP
connection reverse traffic
Configure Contracts for all EPG in a VRF (vzAny)
• vzAny represents the collection of EPGs that Tenant

belong to the same VRF, including L3 VRF1
external. BD1
• Instead of associating contracts to each EPG1

individual EPG you can configure a contract vzAny
to the vzAny BD2
EPG2
• With cross-VRF contracts, vzAny can be a
consumer, not provider
Tenant Shared Tenant Shared
Tenant ONE Tenant ONE
Services Services
EPG1
EPG1 VRF1 VRF Services
VRF1 VRF Services
EPG shared
EPG shared
vzAny service
vzAny service EPG2
EPG2
NOT “SUPPORTED”
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE
Create ACLs per Switch/Port

Specify the type of traffic you want each switch to allow
HSRP/VRRP Collapsed Core

Gateways
802.1q
Access Layer
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
4
Create Contracts
Create ACLs per Switch/Port We will create a Contract to specify how 2 EPGs may talk between
Specify the type of traffic you want each switch to allow each other. This contract will be pushed to the whole fabric (physical,
virtual, etc.) consistently. NO complex IP + Ports to specify like ACLs
HSRP/VRRP Collapsed Core Spine Layer

Gateways
Contract SAP_POLICY VXLAN
802.1q Filters
permit icmp Leaf Layer
Anycast GW
permit tcp eq 80
Access Layer (Bidirectional)
BD 1.1.1.1 BD 2.2.2.1
EPG EPG
Netweaver HANA
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
4
Create Contracts
Create ACLs per Switch/Port We will create a Contract to specify how 2 EPGs may talk between
Specify the type of traffic you want each switch to allow each other. This contract will be pushed to the whole fabric (physical,
virtual, etc.) consistently. NO complex IP + Ports to specify like ACLs
HSRP/VRRP Collapsed Core Spine Layer

Gateways
Contract SAP_POLICY VXLAN
802.1q Filters
permit icmp Leaf Layer
Anycast GW
permit tcp eq 80
Access Layer (Bidirectional)
BD 1.1.1.1 BD 2.2.2.1
EPG EPG
Netweaver HANA
Types of Fabric Routes
MP-BGP
overlay-1
MP-BGP
overlay-1
E1 E2
BD-B1 BD-B2
subnet subnet
int-S1 int-S2
• Internal Routes: Subnets defined under BD are internal routes and create static pervasive routes
within the fabric.
MP-BGP
overlay-1
L3Out-1 L3Out-2
E1 E2 ext
ext
1 BD-B1 BD-B2 2
subnet subnet subnet subnet
ext-S1 int-S1 int-S2 ext-S2
within the fabric.
• External Routes: Routes learned via a routing protocol or static routes configured under an L3Out.
These routes are redistributed into MP-BGP and advertise to all leafs that contain the VRF
Types of Fabric Routes Ensure BGP RR
is configured to
enable MP-BGP
MP-BGP
overlay-1
L3Out-1 L3Out-2
E1 E2 ext
ext
1 BD-B1 BD-B2 2
within the fabric.
• External Routes: Routes learned via a routing protocol or static routes configured under an L3Out.
These routes are redistributed into MP-BGP and advertise to all leafs that contain the VRF
• Transit Routes – Routes advertised between L3Outs.
Types of Fabric Routes – Internal Routes
MP-BGP
overlay-1
L3Out-1 L3Out-2
E1 E2 ext
ext
1 BD-B1 BD-B2 2
There are three requirements to advertise Internal Routes out an L3Out:
MP-BGP
overlay-1
L3Out-1 L3Out-2
1
E1 E2 ext
ext
1 BD-B1 BD-B2 2

1. The BD must be associated with the L3Out*
The association adds prefix entry to route map controlling advertised routes
MP-BGP
overlay-1
L3Out-1 L3Out-2
1
E1 E2 ext
ext
1 BD-B1 BD-B2 2 2
subnet subnet subnet C1 subnet

2. A contract must exists between an EPG within the BD and an external EPG on the L3Out.
The contract creates internal BD route on border leaf (cannot advertise route until it exists locally)
Subnet: int-S2 3
Scope: MP-BGP
❑ Private to VRF overlay-1
❑ Advertise Externally
❑ Share Between VRFs
L3Out-1 L3Out-2
1
E1 E2 ext
ext
1 BD-B1 BD-B2 2 2

3. The subnet must have a public scope (Advertise Externally)
Subnet: int-S2 3 Subnet int-S2 installed on border leaf
Scope: when creating contract
MP-BGP between EPG
❑ Private to VRF E2 and external overlay-1
EPG ext2
❑ Advertise Externally
❑ Share Between VRFs
L3Out-1 L3Out-2
1
E1 E2 ext
ext
1 BD-B1 BD-B2 2 2

3. The subnet must have a public scope (Advertise Externally)
Types of Fabric Routes – External Routes
MP-BGP
overlay-1
L3Out-1 L3Out-2
ext ext
1 2
subnet subnet
ext-S1 ext-S2
MP-BGP
overlay-1
ext-S1
L3Out-1 L3Out-2
ext ext
1 2
subnet subnet
ext-S1 ext-S2
ext-S1
ospf, eigrp, static route RD: L1:V1
redistributed into bgp and RT: ASN:V1 MP-BGP
exported into mp-bgp
overlay-1
ext-S1
L3Out-1 L3Out-2
ext ext
1 2
subnet subnet
ext-S1 ext-S2
• External Routes from ospf, eigrp, or static are redistributed on the border leaf into the local bgp
process.
ext-S1
overlay-1
ext-S1
L3Out-1 L3Out-2
VRF-V1 VRF-V1 VRF-V1

ext import RT from mp- ext
ext-S1 ext-S1
1 bgp and install route 2
subnet via:L1(bgp) into vrf as bgp learn via:L1(bgp) subnet
ext-S1 ext-S2
process.
• The bgp route is exported into MP-BGP with a route-target (RT) of the corresponding VRF. Each leaf
in the fabric with the VRF present will import the RT and install the route. External routes on the non-
originating border leaf will be seen as bgp learned routes.
ext-S1
overlay-1
ext-S1
L3Out-1 L3Out-2

ext import RT from mp- ext
ext-S1 ext-S1
1 bgp and install route 2
subnet via:L1(bgp) into vrf as bgp learn via:L1(bgp) subnet
ext-S1 ext-S2
process.
• The bgp route is exported into MP-BGP with a route-target (RT) of the corresponding VRF. Each leaf
in the fabric with the VRF present will import the RT and install the route. External routes on the non-
originating border leaf will be seen as bgp learned routes.
• External Routes are controlled via Import Route Control flag
Types of Fabric Routes – Transit Routes
MP-BGP
overlay-1
ext-S1
ext-S1
L3Out-1 L3Out-2

ext ext
ext-S1 ext-S1
1 2
subnet via:L1(bgp) via:L1(bgp) subnet
ext-S1 ext-S2
• In this example, external route ext-S1 is a Transit Route when advertised out L3Out-2.
• If OSPF or EIGRP on L3Out-2, ext-S1 is redistributed from BGP into the IGP and advertised.
• Transit Routes are controlled via Export Route Control flag
L3 External Subnet Review
• External Subnets for the External EPG (Security Import)

Used to classify dataplane packets into external EPG for policy enforcement
• Export Route Control - filter Transit Routes advertised out of the fabric.
• Import Route Control - filter External Routes received on an L3Out
• Shared Security Import

used to classify dataplane packets into external EPG for policy enforcement for shared/leaked prefixes
• Shared Route Control

Allows external route to be leaked into another VRF
• Aggregate Export - allows prefixes to be aggregated together in export direction (0/0 or ::/0 only)
• Aggregate Import - allows prefixes to be aggregated together in import direction (0/0 or ::/0 only)
• Aggregate Shared Route - allows prefixes to be aggregated together for shared route control
Cisco APIC and Transit Routing

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE
Configure IP Routing
Configure the routing protocol you may need on each
switch/router to learn routes coming from the outside
OSPF Router

Gateways
802.1q
Access Layer
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
5
Configure IP Routing Create L3 Out
Configure the routing protocol you may need on each Specify on which leaf and port of the fabric you want to enable external
switch/router to learn routes coming from the outside routing. Those routes will be imported inside the ACI Fabric with BGP (auto-
configured) and Spines will serve as Route Reflectors. L3 Outs need a
contract to communicate to EPGs and BDs need to be associated to L3 Outs
OSPF Router

Spine Layer
L3Out Internet
Gateways BGP RRs
Leaf 1 Int 1/15
ospf area 0
802.1q network p2p
mtu ignore Leaf 1
Leaf Layer
IP 221.221.221.2/24 1/15 E1/15 221.221.221.2/24
Access Layer Contract Internet (EPG→L3Out)

permit any (bidirectional) OSPF L3 Out
221.221.221.1
BD 1.1.1.1 Router
EPG
Netweaver
Internet
ACI: What changes?
Easy as 1-2-3-4-5
BEFORE NOW
5
Configure IP Routing Create L3 Out
Configure the routing protocol you may need on each Specify on which leaf and port of the fabric you want to enable external
switch/router to learn routes coming from the outside routing. Those routes will be imported inside the ACI Fabric with BGP (auto-
configured) and Spines will serve as Route Reflectors. L3 Outs need a
contract to communicate to EPGs and BDs need to be associated to L3 Outs
OSPF Router

Spine Layer
L3Out Internet
Gateways BGP RRs
Leaf 1 Int 1/15
ospf area 0
802.1q network p2p
mtu ignore Leaf 1
Leaf Layer
IP 221.221.221.2/24 1/15 E1/15 221.221.221.2/24
Access Layer Contract Internet (EPG→L3Out)

permit any (bidirectional) OSPF L3 Out
221.221.221.1
BD 1.1.1.1 Router
EPG
Netweaver
Internet
Global Settings
Best Practices Summary
• Disable Remote EP Learn

• Prior 3.0: Fabric → Access Policies → Global Policies → Fabric Wide Setting Policy
• After 3.0: System → System Settings → Fabric Wide Setting
• Enforce Subnet Check (will only work EX and FX based leafs)

• Prior 3.0: Fabric → Access Policies → Global Policies → Fabric Wide Setting Policy
• After 3.0: System → System Settings → Fabric Wide Setting
• IP Aging should be enabled

• Prior 3.0: Fabric → Access Policies → Global Policies → IP Aging Policy
• After 3.0: System → System Settings → Endpoint Control → IP Aging
• MCP (per VLAN) should be enabled

• Prior 3.0: Fabric → Access Policies → Global Policies → MCP Instance Policy default
• After 3.0: Fabric → External Access Policies → Policies → Global → MCP Instance Policy default
ACI Fabric Endpoint Learning Evolution
Endpoint Learning Optimization Options
Unicast GARP-based Limit IP L4-L7 Endpoint Disable Remote Enforce IP Data

IP Aging
Routing EP Move Learning Virtual Dataplane EP Learn Subnet Plane
Detection To Subnet IPs Learning (on border leaf) Check Learning
• Avoid using default policy if possible

• Create multiple separate policy so easier to make changes
• Make sure proper naming conventions are used
ACI Fabric Endpoint Learning Whitepaper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html
Forwarding Flow
Drop and ARP
Forward to Does spine knows glean
remote leaf Dest IP in coop ? For destination
No
Yes IP
Summary Forward to
Yes
Does spine knows
No
Drop
Flooded frame remote leaf DMAC in coop ?
reached re mote
leaf
Forward to
Flood within Border Leaf
Forward to Forward to Forward to Forward to Drop
Spine Proxy Spine Proxy Per routing-
local port remote leaf BD (GIPo) local port remote leaf
table
Yes No
Hardware Does Leaf know
Flood
Proxy Dst IP as
L3OUT Routes?
Yes Does Leaf have

No
BD Subnets for Dst IP
Is Dst MAC on What is BD config? Is Dst IP on
(route to Acast Spine)
Local Leaf? (L2 Unknown Uni) Local Leaf?
Yes No Yes No
Does Leaf know

Does Leaf know Dst IP as
Yes Dst MAC? Yes EndPoint?
No No
L2 or L3 ?
L2 L3
(DMAC != ACI MAC) (DMAC == ACI_MAC)
Packet coming to the

Leaf TECDCN-2002 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 356
ACI Policy Model Evolution
MAP EP to EPG
VLAN/VXLAN=EPG IP Based EPG MAC Based EPG VM Attributed based EPG AD / DNS based EPG
Contract Creation and Enforcement
Enforced vzAny Intra-EPG Ingress Contract Contract Intra-EPG Blacklist

Or Unforced Isolation Policy Preferred Group Inheritance Contract (3.2)
(2.2) (2.3) (3.0)
ACI: How do I start?
Easy as 1-2-3-4-5
5) Once all servers are migrated
Your new ACI Fabric to the ACI Fabric, you may
remove your old gear
Internet/WAN
Contract
Nexus 9000
Spine Layer
VXLAN
Nexus 9000
APIC Cluster
EPG 1
EPG 2
1 1.1.1.0/24 2.2.2.0/24
Nexus 7K/5K and legacy Simplify & secure Integrate

Non-disruptive At your own pace
networking migration your DC network virtual & cloud
ACI: How do I start?
Easy as 1-2-3-4-5
Your new ACI Fabric
Internet/WAN If you add more leaves or spines,

Contract APIC will auto-discover and auto-
configure them. It is that SIMPLE!
Nexus 9000
Spine Layer
VXLAN
Nexus 9000
APIC Cluster
EPG 1
EPG 2
1 1.1.1.0/24 2.2.2.0/24
Nexus 7K/5K and legacy Simplify & secure Integrate

Non-disruptive At your own pace
networking migration your DC network virtual & cloud
ACI Software Release Guidelines
Long Lived Releases
1 Two Long Lived Releases At Any Given Point of Time
2 Active Maintenance Will Be Primarily Focused On Long Lived Release
3 Target Duration Of Long Lived Release Support: Up to 18 Months From FCS
4 Direct Upgrade From One Long Lived To Next Long Lived Release Will Be Supported
5 Long Lived Releases Are Recommended For Networks That Will Not be Upgraded Frequently
Short Lived Releases
1 No Active Maintenance Beyond Six Months From FCS
ACI Software Release Cadence
ACI ACI ACI ACI ACI

ACI
3.2(1) 4.1(1) 4.2(1) 5.0(1) 5.1(1)
4.0(1)
Q3CY Q4CY Q1CY Q2CY Q3CY Q4CY Q1CY Q2CY Q3CY Q4CY
Q2CY
18 18 19 19 19 19 20 20 20 20
18
ACI ACI
ACI ACI ACI ACI ACI ACI
4.2(2) 4.2(3)
3.2(2) 4.0(2) 4.1(2) 4.2(4) 5.0(2) 5.1(2)
ACI
3.2(9)
Long Lived Major Releases Major Releases Minor Releases

ACI Software Upgrade Guidelines
Fabric Upgrade Flexibility
1 Centrally manage the fabric upgrade from the APIC
2 Schedule the firmware upgrade for a later date and time
3 Divide the switches into multiple maintenance groups, and upgrade by group
4 Multi-Pod Upgrade: Maintenance groups can be created across different POD
5 Mixed OS operation - Can have two different releases in the fabric at any given time
5 Compatibility check feature in APIC verifies upgrade path between current and new version
ACI Anywhere
• Operational Simplicity: Same Containers Hypervisor
“look and feel” as On-Premise

• Automated Policy Translation:
ACI Anywhere
Consistency across the entire
data center
• Common Governance: Cloud
Exchange
End-to-end discovery, visibility Data
Center
and troubleshooting
On Premises
Cloud
IOT Edge
Cisco ACI Multi-Site Orchestrator
Cisco Virtual ACI (Virtual Edge)
Cisco ACI Cisco ACI Physical Remote Leaf

Key Components
Cisco Virtual ACI (Virtual Pod)
Cisco Cloud ACI
ACI Anywhere – Accelerate Multicloud
“Evolving our multicloud journey by extending ACI everywhere”
ACI Multi-POD
Multiple Networks
(Pods) in a single
Availability Zone
(Fabric)
ACI 2.0
ACI Multi-POD
Multiple Networks
(Pods) in a single
Availability Zone
(Fabric) ACI 3.0
ACI 2.0
ACI Multi-Site
Multiple Availability
Zones (Fabrics) in a
Single Region ’and’
Multi-Region Policy
Management
ACI Multi-POD ACI Remote-Leaf

Multiple Networks Physical Remote Leaf
(Pods) in a single extends an Availability
Availability Zone Zone (Fabric) to
(Fabric) ACI 3.0 remote locations
ACI 2.0 ACI 3.1

ACI Multi-Site
Multiple Availability
Zones (Fabrics) in a
Single Region ’and’
Multi-Region Policy
Management
ACI Multi-POD ACI Remote-Leaf

Multiple Networks Physical Remote Leaf
(Pods) in a single extends an Availability
Availability Zone Zone (Fabric) to
(Fabric) ACI 3.0 remote locations ACI 4.0
ACI 2.0 ACI 3.1

ACI Multi-Site Virtual ACI
Multiple Availability Virtual POD extends an
Zones (Fabrics) in a Availability Zone (Fabric)
Single Region ’and’ to remote locations on
Multi-Region Policy standard VMs
Management
NEW !
ACI Multi-POD ACI Remote-Leaf Cloud ACI
Multiple Networks Physical Remote Leaf ACI Extensions to
(Pods) in a single extends an Availability AWS and Azure
Availability Zone Zone (Fabric) to Public Cloud
(Fabric) ACI 3.0 remote locations ACI 4.0
ACI 2.0 ACI 3.1 ACI 4.1- 4.2

ACI Multi-Site Virtual ACI
Multiple Availability Virtual POD extends an
Zones (Fabrics) in a Availability Zone (Fabric)
Single Region ’and’ to remote locations on
Multi-Region Policy standard VMs
Management
ACI Multi-Site
Multi-Site Orchestrator Site N
(MSO)
3 VM Cluster
VM VM VM VM VM VM VM
Any Routed IP Network
Site1 Site 2
VM VM VM VM VM VM VM VM VM VM VM VM VM VM
No Multicast <= 1s RTT Required (MSO → APIC) Single central management (MSO)
Phased Changes (Zones) Up to 12 Sites, distributed gateway Automated L2 DCI VXLAN extension
ACI Multi-Site
Software and Hardware Requirements
• Support all ACI leaf switches (1st Gen, EX, FX, FX2) Any Routed Network
• Modular Spine with EX/FX line card to

connect to the inter-site network
Can have only a subset
1st Gen 1st Gen -EX -EX of spines connecting to
• Nexus 9364c or Nexus 9332x fixed spine the IP network
supported for Multi-Site from ACI 3.1
release (shipping)
• 1st generation spines (including
Nexus 9336PQ) not supported
• Can still leverage those for intra-site leaf
to leaf communication
ACI Anywhere Shipping
Encrypted DCI Connectivity

Multi-Site
IP / WAN
VM VM VM
Site A Site B Site C

Multi-Site
IP / WAN
MACSEC MACSEC
Today
VM VM VM

Multi-Site
IP / WAN
CloudSec
MACSEC MACSEC
Today
VM VM VM

Multi-Site
IP / WAN
CloudSec
MACSEC MACSEC
Today Future
VM VM VM
ACI Remote Leaf Remote Location A
RL
Any Routed IP Network
Satellite DC
Remote Location B
RL
Pod 1 VM
Brownfield
Remote Location C
RL
Telco/Co-lo
Zero Touch Auto <= 300 ms RTT Required Single central management
Discovery of Remote Leaf Up to 20 Remote Locations Automated L2 VXLAN extension
ACI Remote Leaf Requirements
Hardware & Software
ACI Main DC Remote Location

Supported Spines Supported Leaf
• N93180YC-EX
Fixed Spine • N93108TC-EX
• N9364C • N93180LC-EX
• N9332C (ACI 4.0) • N93180YC-FX
Modular Spine (C9504/C9508/C9516) • N93108TC-FX
• N9732C-EX • N9348GC-FXP
• N9736C-FX • N9336C-FX2
All hardware from –EX onwards is supported
ACI Anywhere: ACI Virtual Edge
Decoupled From Hypervisor Kernel APIs
Multi-Site Orchestrator
Data Center 1
Data Center 2
(ACI Site 1) IPN
IP Network
(ACI Site 2)
Nexus 9000 Nexus 9000

(DC Network) VXLAN (DC Network)
L2 Extension
WAN
Local Router
VM
ACI Virtual Edge VM
Nexus 9000
VM VM VM VM VM VM VM (Remote Leaf Network)
Policy Consistency Across Enable Migration From Maintain Existing Operational

Multiple Hypervisors Legacy To ACI Models
ACI Anywhere: ACI Virtual Pod
Extend ACI To Bare-metal Clouds, Remote Data Centers, and Legacy Infrastructure
Data Center 1
Data Center 2
(ACI Site 1) IPN
IP Network
(ACI Site 2)
Nexus 9000 Nexus 9000

(DC Network) VXLAN (DC Network)
L2 Extension
ACI Virtual Pod 1 ACI Virtual Pod 2
WAN
Local Router
Pod 1 Pod 2
VM
ACI Virtual Edge VM
ACI Virtual Edge ACI Virtual Edge
Nexus 9000
Virtual Spine/Leaf Functionality Single central management

Up to 64 AVEs per vPod
w/AVE integration Automated L2 VXLAN extension
ACI vPod Requirements
Hardware & Software
On-Premises Data Center vPoD Data Center

Supported Spines • VMware vCenter running 6.0 or later
Fixed Spine • 2 hosts for Management cluster
• N9364C
• N9332C
• Management cluster may exist on the
same AVE ESXi nodes
Modular Spine (C9504/C9508/C9516)
• N9732C-EX with N9K-C950x-FM-E(2) • ESXi 6.0 or 6.5
• Each vSpine (x2) & vLeaf(x2) VM consumes 4vCPU,
• N9736C-FX with N9K-C950x-FM-E(2) 16 GB RAM and 80 GB storage
APIC Controller Software • Each AVE (one per ESXi host) VM consumes
2vCPU, 8 GB RAM and 8 GB storage
• ACI 4.0+ onward release
Challenges in building a Multi Cloud environment
Build an automated and Maintain consistent policy, Provide a single pane of

secure interconnect between security and analytics for glass to manage policies
on-Premises and Cloud workloads deployed across across on-premise and cloud
datacenters with ease of on-premises and cloud locations
provisioning and monitoring locations
at scale
Traditional Data Center AWS Azure
Firewall Security Groups ASG
Access Control Lists (ACLs) Security Network ACLs (NACL) NSG
Administrators Identity & Access Management (IAM) Active Directory
Router / Host Routers (CSR1kv) Virtual Private Cloud (VPC) Virtual Network
Switch Networking Gateways (VGW,IGW, TGW) Gateways
Load Balancer Elastic Load Balancing (ELB) ALB
On-Premise Servers Amazon Machine Image (AMI) VHD

Virtual Machines (VM) Servers / VM
Amazon EC2 Instances
Containers Compute AKS
Elastic Container Service (EKS)
SAN Elastic Block Store (EBS), Storage Encryption

Storage & Azure Storage
NAS, NFS Elastic File System (EFS), S3
Databases Azure SQL
RDBMS Amazon RDS
ACI Anywhere: Public Cloud Extensions
Seamlessly Connect Multiple Data Centers
Data Center 1
Infra VPC
(ACI Site 1) IPN
IP Network
Nexus 9000
(DC Network) VXLAN User VPC User VPC
L2 Extension
WAN
Local Router
VM
ACI Virtual Edge VM
Nexus 9000
Data Center 1
Infra VPC
(ACI Site 1) IPN
IP Network
Nexus 9000
L2 Extension
WAN
Local Router
VM EPG
Web ACI Virtual
Contract
APPEdge
EPG
Contract VMEPG
DB
SG
Web
SG Rule
SG
APP
SG Rule
SG
DB
Nexus 9000
Data Center 1
Infra VPC
(ACI Site 1) IPN
IP Network
Nexus 9000
L2 Extension
WAN
Local Router
VM EPG
Web ACI Virtual
Contract
APPEdge
EPG
Contract VMEPG
DB
SG
Web
SG Rule
SG
APP
SG Rule
SG
DB
Nexus 9000
Discovery Policy CSR-1Kv/Direct- Single Point Operational

& Visibility Translation Connect integration Of Orchestration Consistency
Data Center 1
Infra VPC
(ACI Site 1) IPN
IP Network
Nexus 9000
L2 Extension
WAN
Local Router
VM EPG
Web ACI Virtual
Contract
APPEdge
EPG
Contract VMEPG
DB
SG
Web
SG Rule
SG
APP
SG Rule
SG
DB
Nexus 9000

Data Center 1
Infra VPC
(ACI Site 1) IPN
IP Network
Nexus 9000
Encrypted L2 Extension
Encrypted
WAN
Local Router
VM EPG
Web ACI Virtual
Contract
APPEdge
EPG
Contract VMEPG
DB
SG
Web
SG Rule
SG
APP
SG Rule
SG
DB
Nexus 9000

Cloud ACI
Cloud ACI
1 Universal Policy and Operational Model for MultiCloud
Cloud ACI
2 Network Automation across On-Premises and MultiCloud
Cloud ACI
3 Uniform Segmentation Policy for On-Premises and Cloud
Cloud ACI
Automated life cycle management of CSR1KV & Cloud

4 Resources
Cloud ACI

4 Resources
5 Extensible, Elastic Software Architecture
Cloud ACI

4 Resources
End-to-End visibility, Monitoring, Troubleshooting and Common

6 governance
Cloud ACI

4 Resources
End-to-End visibility, Monitoring, Troubleshooting and Common

6 governance
7 Foundation for Multi Domain Policy Connectivity
Virtual Private Network (VPN)
Multisite Orchestrator
Public Cloud Site B

Site A On-Premise
User VPC-1
VGW
CSR1000V
Customer
Premise AWS
AWS Instances
Internet
Router Internet
Gateway
Infra VPC VGW
VM VM VM AWS Instances
User VPC-2
AWS Region
Public Cloud Site B

Site A On-Premise
User VPC-1
IPSec VPN Tunnel (Underlay)
VGW
CSR1000V
Customer
Premise AWS
AWS Instances
Internet
Router Internet
Gateway
Infra VPC VGW
User VPC-2
AWS Region
Public Cloud Site B

Site A On-Premise
User VPC-1
VGW
CSR1000V
VXLAN Tunnel (Data Plane)

Customer
Premise AWS
AWS Instances
Internet
Router Internet
Gateway
Infra VPC VGW
User VPC-2
AWS Region
Public Cloud Site B

Site A On-Premise
User VPC-1
VGW
CSR1000V
BGP-EVPN Session (Control Plane)

Customer
Premise AWS
AWS Instances
Internet
Router Internet
Gateway
Infra VPC VGW
User VPC-2
AWS Region
Public Cloud Site B

Site A On-Premise
User VPC-1
VGW
CSR1000V
BGP-EVPN Session (Control Plane)

Customer
Premise AWS
AWS Instances
Internet
Router Internet
Gateway
Infra VPC VGW
User VPC-2
AWS Region
Direct Connect (DX)
On-Premise
Site A Public Cloud Site B
User VPC-1
VXLAN VGW
BGP-EVPN
Direct Connect (DX) / BGP Underlay CSR1000V AWS Instances
Border Amazon
ACI Leaf DGW/
VGW Infra VPC
VM VM VM
AWS Region VGW
• Direct Connect and BGP underlay between Infra-VPC and

ACI Border Leaf AWS Instances
• BGP-EVPN and VXLAN over Direct Connect ACI fabric to User VPC-2
CSR 1000v
Direct Connect (DX)
On-Premise
Site A Public Cloud Site B
User VPC-1
VXLAN VGW
BGP-EVPN
Direct Connect (DX) / BGP Underlay CSR1000V AWS Instances
Border Amazon
ACI Leaf DGW/
VGW Infra VPC
VM VM VM
AWS Region VGW
• Direct Connect and BGP underlay between Infra-VPC and

ACI Border Leaf AWS Instances
• BGP-EVPN and VXLAN over Direct Connect ACI fabric to User VPC-2
CSR 1000v
Policy Mapping - AWS For your info
& reference
User Account Tenant

Virtual Private Cloud VRF
VPC subnet BD Subnet
Tag / Label EP to EPG Mapping
Security Group EPG

Network Access List Taboo
Security Group Rule Contracts, Filters
Outbound rule Consumed contracts
Source/Destination: Subnet or IP or Any or ‘Internet’
Protocol
Port
Inbound rule Provided contracts
EC2 Instance
Network Adapter End Point (fvCEp)
Policy Mapping - Azure For your info
& reference
Resource Group Tenant

Virtual Network VRF
Subnet BD Subnet
Application Security Group EPG

(ASG)
Network Security Group

(NSG) Filters
Outbound rule Consumed contracts

Source/Destination: ASG or Subnet or IP or Any or ‘Internet’
Protocol
Port
Inbound rule Provided contracts

Virtual Machine
Network Adapter
Cloud Hierarchy For your info
& reference
ACI AWS Azure GCP

Security Domain OU Organization Organization
Tenant Account Subscription Project
Site / Pod Region Region VPC
VRF / Context VPC VNet Region
Path / Node Availability
AZ Subnet
Attachment sets/zones
BD Subnet Subnet Subnet Zone
Virtual Networking Comparison For your info
& reference
AWS Azure GCP

VPC is regional VNet is regional VPC is global
Dedicated HW option Some large mem VM Sole tenant
IPv4/IPv6 IPv4/IPv6 IPv4/ IPv6 proxy
Internet access by Internet access by
Public private subnets
default default
CIDR can’t change/new Add, remove, expand or
CIDR can expand only
blocks can be added shrink
Subnet in AZ Subnets are regional Subnets are regional
Use case #1: Hybrid-Cloud Deployment
Region(s) On-Premises Region(s)
Hybrid Cloud supported with AWS in Q1-CY19 and Azure in Q2-CY19
Use case #2: Cloud First with Multiple Regions
Cloud APIC
US-West Region London Region Seoul Region
One ACI Policy Domain with Multiple AWS Regions
Use case #3: Multi-Cloud
Region(s) Region(s) Region(s)
Multi-Cloud with AWS and Azure Cloud Sites supported in 2H-CY19

Operations
APIC Management Information Model Reference
From APIC GUI
https://apic/doc/html/
Visore – Web Based MO Query and Browser Tool
https://<IP>/visore.html
Network Monitoring and Troubleshooting Tools
Physical Network Abstracted Network

• ping • properties (EP / TEP / contract)
• traceroute • health scores / faults / events / audit
• show (interface / table / etc) • iping, itraceroute

• atomic counters
• syslog
• statistics
• SPAN
• diagnostics (on-demand)
• SPAN
• ELAM
Capacity Dashboard
Configuration Rollback
Endpoint Tracker
Traffic Map - Visualization
Help visualize and quickly spot high traffic density and underutilized
nodes in the Cisco ACI™ fabric.
A grid is presented with a list of node IDs or vPC pairs on each axis.
Traffic flow between a given pair of nodes or between a vPC pair is
presented using color-coded cells on the heat map.
Traffic density is presented in a range of colors, from lightest

(yellow), to shades of orange, to red (highest). Traffic statistics are
collected using atomic counters.
• You can order by name or by traffic.

• Traffic can be seen by:
- Sent packets
- Received packets
- Dropped packets
- Excess packets
What is Ftriage
ACI Debugging
• Fabric triaging tool. Python utility, runs on APIC in admin mode

• Logs into switch nodes to capture requested data with commands/query/ELAM
• Driven by specific user inputs which are validated first
• Runs Elam to determine packet data path
• Traces packet hop by hop till the point where it exits the fabric or gets dropped
• Drop reason is provided along the node, interface info
Usability Enhancements 4.1
Unified Reskin
Alert List
• Alert to detect if OSPF connectivity

is down (MPoD) configuration
• Alert to detect process crash and

acknowledge old crashes
Interface Status & Favorite Tab
• This status is shown only for physical interfaces prior to 4.1,

now all other interfaces like VPC / PC will have the status too
• User can mark any tab as their favorite tab and they will be navigated to that tab every
time the policy load
First Time Setup Wizard
4.2
Configuration
• Help onboard new customers with basic

features
• Leaf & Spine provisioning. BGP route
reflector configuration, NTP, DNS, Out-
of-Band management all can be
configured from one place
• Suggest best practices for new ACI
deployments
• Ability to review existing configuration
Topology Improvements
4.2
Operations
4.2
Operations
4.2
Operations
ACI 4.2: Usability Enhancements 4.2
Simplify L3Out in 3 Steps
Cisco ACI App Center Apps
Programmable Infrastructure: Open API’s for Value Added Applications
https://aciappcenter.cisco.com/
Matthias Wessendorf
ACI Technical Marketing Engineer
Programmability
The APIC REST API is the Core of ACI
Programmability
Programmability
REST API
Programmability
GUI
REST API
Programmability
GUI CLI
REST API
Programmability
GUI CLI Python
REST API
ACI Object Model
ACI Object Model
• Objects within APIC are structured in

tree-based hierarchy
ACI Object Model

• Objects are referred to as Managed

Objects (MO)
ACI Object Model


Objects (MO)
• Every object has a parent, with

exception of top:Root (top of tree)
ACI Object Model


Objects (MO)
• Every object has a parent, with

exception of top:Root (top of tree)
• Relationships exist between objects
How to Identify Objects
Distinguished Name
fvAp fvAEPg
polUni fvTenant vzFilter vzEntry
vzBrCP vzSubj
topRoot fabricPathEpCont fabricPathEp
fabricTopology fabricPod
fabricNode
vmmProvP vmmDomP vmmCtrlrP
Distinguished Name
fvAp fvAEPg
vzBrCP vzSubj
fabricNode
EPG in tenant “Cisco” under application “DNS”

uni/tn-Cisco/ap-DNS/epg1
Distinguished Name
fvAp fvAEPg
vzBrCP vzSubj
fabricNode
EPG in tenant “Cisco” under application “DNS” Interface Eth1/4 on leaf 102 in pod 1
uni/tn-Cisco/ap-DNS/epg1 topology/pod-1/paths-102/pathep-[eth1/4]
The REST API Exposes the Object Model
http(s)://
http or
https
protocol
http(s):// host:port
http or
APIC host
https
and port
protocol
http(s):// host:port /api
http or
APIC host API
https
and port Operator
protocol
http(s):// host:port /api /{mo|class}
http or Specify
APIC host API
https Managed
and port Operator
protocol Object or Class
Operator
http(s):// host:port /api /{mo|class} /{dn|classname}
http or Specify Distinguished

APIC host API
https Managed name or Object
and port Operator
protocol Object or Class Class
Operator
http(s):// host:port /api /{mo|class} /{dn|classname} .{xml|json}
http or Specify Distinguished

APIC host API Encoding for
https Managed name or Object
and port Operator response
protocol Object or Class Class
Operator
http(s):// host:port /api /{mo|class} /{dn|classname} .{xml|json} ?[options]
http or Specify Distinguished Specify filters,

https Managed name or Object selectors or
protocol Object or Class Class modifiers to query,
Operator joined using
ampersand (&)

ampersand (&)
Read properties for an EPG by Distinguished Name:
http://apic/api/mo/uni/tn-Cisco/ap-Software/epg-Download.xml

ampersand (&)
Read properties for an EPG by Distinguished Name:
http://apic/api/mo/uni/tn-Cisco/ap-Software/epg-Download.xml
Find all 10G ports on fabric:
http://apic/api/class/l1PhysIf.xml?query-target-filter=eq(l1PhysIf.speed,"10G")
ACI Python SDK
ACI Python SDK, AKA Cobra
• Cobra is a native Python API library that abstracts the APIC REST API
• Supports lookups, creations, modifications, deletions
• Objects in Cobra are a 1:1 representation of objects in the ACI object model
• All data has client side consistency checks performed
• Packaged as .egg, install with easy_install
• Packaged as .egg, install with easy_install
from cobra.model.fv import Tenant
from cobra.model.pol import Uni
from cobra.mit.request import ConfigRequest
uniMo = Uni('')
t = Tenant(uniMo, 'Tenant1') # We create a tenant as a child of the universe
c = ConfigRequest() # Create a ConfigRequest to contain our new object
c.addMo(t) # Add our tenant to the ConfigRequest
moDir.commit(c) # Commit our configuration request
Simple 3-Tier App with Cobra
from cobra.model.fv import *

from cobra.model.pol import Uni
uniMo = Uni('')
t = Tenant(uniMo, 'Tenant1')
ap = Ap(t, 'Exchange')
epg1 = AEPg(ap, 'OWA')
epg2 = AEPg(ap, 'FrontEnd')
epg3 = AEPg(ap, 'MailBox')
ep = RsPathAtt(epg1, tDn =‘topology/pod-1/paths-17/paths-[eth1/1]’,
mode=‘regular’, encap =‘vlan-10’)
c = ConfigRequest()
c.addMo(t)
moDir.commit(c)
Simple 3-Tier App with Cobra
from cobra.model.fv import *

from cobra.model.pol import Uni Underlay and overlay
automation with a single API
uniMo = Uni('') model
t = Tenant(uniMo, 'Tenant1')
ap = Ap(t, 'Exchange')
epg1 = AEPg(ap, 'OWA')
epg2 = AEPg(ap, 'FrontEnd')
epg3 = AEPg(ap, 'MailBox')
ep = RsPathAtt(epg1, tDn =‘topology/pod-1/paths-17/paths-[eth1/1]’,
mode=‘regular’, encap =‘vlan-10’)
c = ConfigRequest()
c.addMo(t)
moDir.commit(c)
Ansible for ACI
Ansible for ACI
GUI CLI Python
REST API
Ansible for ACI
GUI CLI Python
REST API
You can create
arbitrarily
complex/rich items.
This example shows
how to use a single
play to create
provider or consumer
ACI contracts. No
need to create two
plays (one for
consumer contracts,
one for provider)!
Query ACI
Terraform
What is Terraform?
Infrastructure Configuration
Provisioning Tools Management Tools
Ansible
Terraform
Puppet
CloudFormation
Chef
Heat
Salt Stack
ARM templates
CF Engine
…
…
Terraform introduction
• Terraform is an open-source Infrastructure Provisioning Tool from Hashicorp
• Ansible, Puppet, SaltStack, Chef are Configuration Management Tools
• It is common to combine Terraform and Ansible (or Chef, Puppet, etc.)
• Terraform ships as a single binary ready to run

• Available for Linux, Windows and MacOS
• Zero server-side dependencies, strictly client-side architecture only
• Terraform use a declarative language called HCL (Hashicorp Configuration Language)
• Terraform is extensible by using plugins created by Hashicorp or infrastructure/software vendors

directly
• There is a Cisco ACI plugin (developed by Cisco)
• Plugins for vSphere, AWS, Kubernetes, Openstack, Azure, Google Cloud, etc. also exist
Terraform introduction
• While Ansible can provision servers and Terraform can configure resources, key differences are:
1. Terraform keeps state locally
• it knows what is configured vs desired end-state.
2. Terraform defaults to an immutable infrastructure paradigm
• Ansible relies on a mutable infrastructure paradigm:

• If you tell Ansible to install a new version of Nginx, it will run a software update on your existing servers
• Changes happen in place and each server builds a unique history of changes
• Over time as more and more updates are applied, each server becomes slightly different from others
• You eventually end up with config drift which is not desirable
• Terraform driven changes often result in a completely new server being deployed
• The existing one is simply destroyed and replaced with a new instance
• Hence the infrastructure is said to be immutable
• Terraform knows what’s actually deployed and compares with your declarative instructions
• Terraform can rectify config drift and re-apply well-known configurations
Terraform or Ansible?
• Both Ansible and Terraform can coexist
• It’s not an either/or story
• You can use Terraform to spin up VMs from a template then use Ansible once the VM is up for
example
• Likewise with EC2 instances, containers, etc.
• Terraform is not really meant to configure servers once they’re up

• It does offer local and remote code execution capabilities, but it isn’t its strength
• Those are called provisioners
• It can call Ansible once the server has been provisioned
• Terraform ships with local-exec and remote-exec provisioners
• Both can be used to run Ansible playbooks, either locally on the Terraform server
• Or directly on the remote machine
Terraform concepts
• Terraform has 3 important building blocks:
• Providers
• Resources
• Read-write resources
• Read-only resources (called data)
• Variables
• Providers describe a type of infrastructure provider

(Cisco ACI, AWS, vSphere, Openstack, K8s, etc.)
• Resources are specific to a given provider.
• Example on the right-hand side with AWS
• Variables help you separate data from logic
• This example does not use separate variables,
they’re directly included with the “code”
• A collection of HCL instructions is called an execution

plan
Infrastructure as Code with ACI
• ACI stores all configuration as a graph database linking objects together (object model)
• ACI object model is a distributed tree structure that is fully accessible through a REST API
• Every node is a managed object (MO) of a specific class, containing attributes and a
distinguished name (Dn – its unique address in the tree)
API
Root
Tenant
Policy
VRF Universe Fabric Hypervisors
10.10.0.1/24 Virtual
BD Tenants VLANs Network
Nodes
EPG EPG
Applications
VLAN 1001 VLAN 1002
ACI Provider Resources
Available Resource Data Source

aci_tenant
aci_vrf
aci_bridge_domain
aci_subnet
aci_application_profile
aci_application_epg
aci_contract
aci_contract_subject
aci_filter
aci_filter_entry
aci_vmm_domain
aci_l3_outside
aci_external_network_instance_profile
aci_leaf_interface_profile

aci_interface_fc_policy
aci_l2_interface_policy
aci_pcvpc_interface_policy_group
aci_leaf_access_port_policy_group
aci_lldp_interface_policy
aci_miscabling_protocol_interface_policy
aci_ospf_interface_policy
aci_access_port_selector
aci_access_port_block
aci_lacp_policy
aci_port_security_policy
aci_leaf_profile
aci_end_point_retention_policy
aci_attachable_access_entity_profile

aci_vlan_encapsulationfor_vxlan_traffic
aci_rest
Connecting to ACI
• You can use username and password or certificate-based authentication
• Certificates based authentication is recommended
path to user private key provider "aci" {

username = "${var.aciUser}"
Name of cert object in ACI private_key = "${var.aciPrivateKey}"
APIC address
cert_name = "${var.aciCertName}"
insecure = true
url = "${var.aciUrl}"
}
Creating a tenant, VRF, BD, EPG
ACI plans make explicit reference to

relations between objects. See the
example here with tenant_dn,
relation_fv_rs_ctx or
bridge_domain_dn
Manage Relation Source Objects
• ACI leverages objects to build relations to other nodes in different part of the tree
• These objects are represented as arguments under the target Terraform resource
• Example with EPG that can have relation to VMM domain (fvRsDomAtt), Bridge-
Domain (fvBD) and Contract consumer/provider (vzBrCP)
“uni/vmmp-VMware/dom-VDS01”
“Web”
“bd1”
“Web”
ACI example
resource "aci_application_epg" "epg2" {
application_profile_dn = "${aci_application_profile.app1.id}"
name = "epg2"
relation_fv_rs_bd = "${aci_bridge_domain.bd1.name}"
relation_fv_rs_dom_att = ["${data.aci_vmm_domain.vds.id}"]
resource "aci_tenant" "demo" { relation_fv_rs_prov = ["${aci_contract.contract_epg1_epg2.name}"]
name = "${var.tenantName}" }
description = "created by terraform"
} resource "aci_contract" "contract_epg1_epg2" {
tenant_dn = "${aci_tenant.demo.id}"
resource "aci_vrf" "vrf1" { name = "Web"
tenant_dn = "${aci_tenant.demo.id}" }
name = "vrf1"
} resource "aci_contract_subject" "Web_subject1" {
contract_dn = "${aci_contract.contract_epg1_epg2.id}"
resource "aci_bridge_domain" "bd1" { name = "Subject"
tenant_dn = "${aci_tenant.demo.id}" relation_vz_rs_subj_filt_att = ["${aci_filter.allow_https.name}","${aci_filter.allow_icmp.name}"]
relation_fv_rs_ctx = "${aci_vrf.vrf1.name}" }
name = "bd1"
} resource "aci_filter" "allow_https" {
tenant_dn = "${aci_tenant.demo.id}"
resource "aci_subnet" "bd1_subnet" { name = "allow_https"
bridge_domain_dn = "${aci_bridge_domain.bd1.id}" }
name = "Subnet" resource "aci_filter" "allow_icmp" {
ip = "${var.bd_subnet}" tenant_dn = "${aci_tenant.demo.id}"
} name = "allow_icmp"
}
resource "aci_application_profile" "app1" {
tenant_dn = "${aci_tenant.demo.id}" resource "aci_filter_entry" "https" {
name = "app1" name = "https"
} filter_dn = "${aci_filter.allow_https.id}"
ether_t = "ip"
data "aci_vmm_domain" "vds" { prot = "tcp"
provider_profile_dn = "${var.provider_profile_dn}" d_from_port = "https"
name = "GFAB1" d_to_port = "https"
} stateful = "yes"
}
resource "aci_application_epg" "epg1" {
application_profile_dn = "${aci_application_profile.app1.id}" resource "aci_filter_entry" "icmp" {
name = "epg1" name = "icmp"
relation_fv_rs_bd = "${aci_bridge_domain.bd1.name}" filter_dn = "${aci_filter.allow_icmp.id}"
relation_fv_rs_dom_att = ["${data.aci_vmm_domain.vds.id}"] ether_t = "ip"
relation_fv_rs_cons = ["${aci_contract.contract_epg1_epg2.name}"] prot = "icmp"
} stateful = "yes"
}
Azeem Suleman
Principal Engineer
Network Assurance &
Insights
Managing the Complete Lifecycle
Day Day Day
0 1 2
Infrastructure Discover, Assure,
planning, delivery configure monitor and
and installation and secure troubleshoot
What’s happening on Day 2 and
what needs to change?
Customer Challenges
Today’s tools do not address modern network needs
Fragmented Reactive Limited Insights

• Too many tools • Inconsistent API • Low data fidelity that is not
addressing silo-ed architecture actionable
visibility use cases • Specialized knowledge • Lack of data correlation,
• Some are old, and some required don’t get the full picture
are expensive • Difficult to root-cause • No dataplane visibility
• Different issue, and often time too
protocols/mechanisms late to react to it
Assure intent
“Ensure the business needs
are consistently maintained”
Cisco Data Center
Network Assurance
and Insights Suite Guarantee Reliability
“Solve problems before they
Transform day 2 networking impact business”
operations from reactive to
proactive
Troubleshoot intelligently
“Highlight the needle in the
haystack”
Day 2 Product Overview
Network Assurance: How It Works
Data Comprehensive Intelligent

Collection Network Modeling Analysis
Captures all non-packet data: Mathematically accurate models 5000+ domain knowledge-based
intent, policy, state across spanning underlay, overlay and error scenarios built-in, codified
data center network virtualization layers remediation steps
User Interface: Centered Around “Smart Events”
Smart Events: What, Where, Why, and How
Change
Management
Incidence and
Change
Problem
Management
Management
Incidence and
Change Compliance and
Problem
Management Visualization
Management
Network Insights: How it Works
Extensions to Fabric Controller
App Store
DCNM APIC
Platform
App Hosting Framework App Hosting Framework
App Store App Store
Data collection and ingestion Data correlation and analysis Data visualization and action
Visibility Insights Proactive Troubleshooting

Learn from your network and See problems before Find root cause faster with
recognize anomalies your end users do granular details
Network Insights Resources
Understand What’s Running In Your Network
System and Resources
Network
Event Analytics
Insights
Resources
Flow Analytics
Deep Insights Into Network Health

(Control Plane, Data Plane, Capacity, Utilization and Environmental Health)
Resource
Event Analytics Dashboard Analytics
Data Collection
Anomaly
Detection
Remediation
Event Analytics Dashboard Displays Faults, Events, And Audit Logs In A Time Series Fashion.
Flow Anomalies Flow Analytics Dashboard
Packet Drops
Latency
End Point Move
Flow Analytics Dashboard Displays Key Indicators Of Infrastructure Data Plane Health.
Network Insights Advisor
Before Network Insights Advisor After Network Insights Advisor
Time to resolve issue in Hours
Notice/Anomaly Detection 4
360 Case Accepted, Outputs Attached

240 Tech Support Analysis 3

120 Back and Forth Communication

120 Back and Forth Communication
120 Remediation, Close case 1
840 Hrs 240 Tech Support Analysis 3

(~35 days) 120 Back and Forth Communication

<1 Day

<1 Day
Downtime/Outages to the Network cost Millions

<1 Day
Success NIA immediately flags anomalies NIA helps prevent Significant OPEX, CAPEX and
metrics and optimizes your network downtimes/outages time savings
360 Case Accepted, Outputs Attached 2

<1 Day
Success NIA immediately flags anomalies NIA helps prevent Significant OPEX, CAPEX and
metrics and optimizes your network downtimes/outages time savings
Dashboard ”Give me a summary of issues”

!
Advisories, Notifications, PSIRTs

• Provide Timely updates about your
system
• Track Bugs and PSIRTs
Anomalies
• Compliance, Consistency, unplanned
events
(Fabric) Fabric wide analysis
Network Insights Resources For your info
& reference
Common Use Cases
& reference
Common Use Cases

Dashboard -- “Tell me now if I’ve got a problem!”
• Anomalies
!
& reference
Common Use Cases

• Anomalies
!
System
• Resource Utilization (fabric wide)
• Trend Monitoring
(rising/falling)
• Fabric Capacity
• Environmental
& reference
Common Use Cases

• Anomalies
!
System
• Resource Utilization (fabric wide)
• Trend Monitoring
(rising/falling)
• Fabric Capacity
• Environmental
Operations
• Statistics
• Flow Analytics
• Event Analytics
Network Insights Advisor For your info
& reference
Notify About Anomalies
Weekly
Sync
NIA
Insight
DB Fabric
& reference
Weekly
Sync
NIA
Insight
DB Fabric
Monitor 1
& reference
Weekly
Sync
NIA
2 Detect
Insight
DB Fabric
Monitor 1
& reference
Known Anomalies
3 Alert / Inform Detected:

CSCDT2396 SAL1820SDRE
Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
2 Detect
Insight
DB Fabric
Monitor 1
& reference
Known Anomalies

Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
2 Detect
Insight
DB Fabric
Monitor 1
4 Implement
& reference
Known Anomalies

Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
2 Detect
Insight
DB Fabric
Monitor 1
4 Implement
& reference
Known Anomalies

Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
2 Detect
Insight
DB Fabric
Monitor 1
4 Implement
Detect
& reference
Known Anomalies

Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
2 Detect
Insight
DB Fabric
Monitor 1
4 Implement
Detect Alert
& reference
Known Anomalies

Weekly
Sync Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
2 Detect
Insight
DB Fabric
Monitor 1
4 Implement
Detect Alert Remediate

& reference
Notify Me About New Releases
s
p
NIA
Push
Insight
Notification
DB Fabric
p PSIRT
s S/W
Notify

& reference
NIA
p
Push s
Insight
Notification
DB Fabric
p PSIRT
s S/W
Notify

& reference
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
p PSIRT
s S/W
Notify

& reference
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
p PSIRT
2 Identify Switches
s S/W p p p
Notify

& reference
Detected:
PSIRT: SAL1820SDRE
3 Alert / Inform
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
p PSIRT
2 Identify Switches
s S/W p p p
Notify

& reference
Notifications
Detected:
PSIRT: SAL1820SDRE
3 Alert / Inform
Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
4 Implement
p PSIRT
2 Identify Switches
s S/W p p p
Notify

& reference
Notifications
Detected:
PSIRT: SAL1820SDRE
3 Alert / Inform
Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
4 Implement
p PSIRT
2 Identify Switches
s S/W p p p
Notify
Detect
& reference
Notifications
Detected:
PSIRT: SAL1820SDRE
3 Alert / Inform
Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
4 Implement
p PSIRT
2 Identify Switches
s S/W p p p
Notify
Detect Alert
& reference
Notifications
Detected:
PSIRT: SAL1820SDRE
3 Alert / Inform
Recommend:
Upgrade S/W to NXOS
7.0(3)I7(3) in SAL1820SDRE
NIA
p
Push s
Insight
Notification
DB Fabric
Monitor 1
4 Implement
p PSIRT
2 Identify Switches
s S/W p p p
Notify

Day 2 Operations in a Life of an
Engineer
Let’s start the Day
with few scenario’s
My Database is
slow!
I can’t reach my
Website!
I need to do
Capacity Planning
Was my Change
successful?
Key Takeaways
• Day-2 Operations is a critical!!
• Network Assurance and
Insights capabilities are going
beyond element managers!
• Telemetry is the enabler!
Conclusions
Network Stack API
Assurance & Insights Deep Visibility
Policy Federation – Cloud

Multi Site & Domain Extension
Fabric Level API’s

Network Semantics
DCNM APIC (DCNM, APIC)
Application Semantics
(APIC)
Fabric (NX-OS) Fabric (ACI)
Device Level API’s
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.
Continue your education
Demos in the
Walk-In Labs
Cisco Showcase
Meet the Engineer

Related sessions
1:1 meetings
Thank you

Tecdcn 2002

Uploaded by

Copyright:

Available Formats

You might also like

Tecdcn 2002

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tecdcn 2002

Uploaded by

Copyright:

Available Formats

Next Generation Data Center

Brenden Buresh – Principal Architect

Thomas Scheibe Brenden Buresh

• Zero Trust / Identity

Application Policy Automation

Application Automation Infrastructure Automation

1. End-end segmentation Compliance monitoring

2. Flexibility to enforce in Cloud 2

1. End-end segmentation Compliance monitoring

2. Flexibility to enforce in Cloud 2

1. End-end segmentation Compliance monitoring

2. Flexibility to enforce in Cloud 2

Network and Policy

• Proactive Guarantee Reliability

Auto Scale AWS API

Auto Scale AWS API

“open = API use “open = anybody

Developer Cloud(s) Data Center

1. Development in Cloud First

Server Silicon Cloud Scale Technology

100G QSFP 50/400G 100/400G

10/40G 25/100G 50/400G 100/400G

28nm 14/16nm 7nm

2017 2018 2019 2020

Pluggable Options Pluggable Options

SFP SFP28 QSFP DD

SFP SFP28 QSFP QSFP28 QSFP DD

SFP SFP28 QSFP QSFP28 QSFP DD

Assurance & Insights Deep Visibility

Policy Federation – Cloud

Fabric Level API’s

Layer-3 HSRP HSRP

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Layer-3 HSRP HSRP

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Layer-3 HSRP HSRP

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Layer-3 HSRP HSRP

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Layer-3 HSRP HSRP

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Layer-3 HSRP HSRP

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

Network Overlays Host Overlays

VTEP VTEP VTEP VTEP

Network Overlays Host Overlays

• Physical and Virtual

Spine Spine Spine Spine

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Spine Spine Spine Spine

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Spine Spine Spine Spine

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Spine Spine Spine Spine

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Spine Spine Spine Spine

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal