BRKDCN 2988

Design, Automate, and
Manage VXLAN BGP EVPN

Multi—Site with NDFC
Parth Patel, Technical Leader, Technical Marketing Engineer:

Data Center and Provider Connectivity BU
BRKDCN-2988
• VXLAN EVPN Multi-Site
Overview
• Design VXLAN EVPN
Multi-Site with NDFC
• Why NDFC?
Agenda • Automate and Manage

VXLAN EVPN Multi-Site
with NDFC
• Conclusion
BRKDCN-2988 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Data Center
Network
Requirements
What are our basic network requirements?
1) Provide paths for endpoints to communicate at
Layer2(MAC) and Layer3(IP)
2) Provide separation of endpoint into Layer2
forwarding domains (VLAN or BD)
EP1 VLAN EP2 EP3 VLAN EP4

10 20
VRF-1
1) Provide paths for endpoints to communicate at
3) Routing between IPv4/IPv6 subnets and allow
separation of these into multiple VRFs
EP1 VLAN EP2 EP3 VLAN EP4

10 20
VRF-1
1) Provide paths for endpoints to communicate at 4) Communication to external L2 networks (DCI)
EP1 VLAN EP2 EP3 VLAN L2

EP4
10 20External
VRF-1
1) Provide paths for endpoints to communicate at 4) Communication to external L2 networks (DCI)
Layer2(MAC) and Layer3(IP) 5) Communication to external L3 networks (WAN)
EP1 VLAN EP2 EP3 VLAN L2

EP4
L3
10 20External External
VRF-1
6) Allow security policies in order to limit communication to between endpoints to allowed protocols.
ip access-list web-in
VLAN 10 VRF1 VLAN 20 permit tcp Subnet1 Subnet2 eq 80
Subnet1 Subnet2 ip access-list web-out
permit tcp Subnet2 eq 80 Subnet1
EP1 EP3
80
ip access-group web1 in
ip access-group web2 out

EP2
22 EP4
What Physical Topology is required?
• Physical topology must support our endpoint communication
(layer-2 / layer-3), and the location of endpoints within the physical
network will affect the supporting design/configuration.
VLAN EP2 VLAN EP4L2 L3

EP1 EP3
10 20 External External
VRF-1
Data Center Network Evolution
Well-Known but Legacy Methods
Core
Layer-2 STP Forwarding
STP Root STP 2nd Root Layer-2 STP Blocked

FHRP Active FHRP Standby
(Aggregation) (Aggregation) Layer-3 ECMP
Access
Classic Spanning-Tree
Core
Layer-2 STP Forwarding
STP Root STP Root

Layer-3 ECMP
FHRP Active FHRP Active
(Aggregation) (Aggregation)
Access
VPC and Spanning-Tree

Core
Layer-2 FabricPath
Anycast HSRP Anycast HSRP

(Spine) Layer-3 ECMP
(Spine)
Leaf
FabricPath (Mac-in-Mac)
Data Center Network Challenges
Of Legacy Methods
Hierarchical Topology Hair-Pining Flood & Learn

• Scale-Up with Big • Suboptimal • Convergence
Centralized Chassis performance, traffic dependent on Single
(Aggregation) forwarding constrained Tree and MAC Flush
by spanning-tree rules (TCN)
• STP limits full bandwidth
utilization • Rigid Network Service • Exposed to Large
Placement (L4-L7) Broadcast Domains (All
Access and
• Limited Endpoint
Aggregation)
Mobility
Next Generation– VXLAN BGP EVPN Fabrics
Becoming Industry De-Facto Standard
• An IGP is recommended for the underlay
(OSPF or IS-IS) R R
• BGP can also be used if needed
• BGP EVPN must be used in the overlay to

exchange endpoints information V V V V
• Spines act as route-reflectors
• VXLAN is used to transport endpoint traffic Underlay

in the fabric BGP EVPN Overlay
• Leafs are considered VTEP as they R BGP Route-Reflector
encapsulate and decapsulate VXLAN
traffic V VXLAN Tunnel Endpoint (VTEP)
Data Center Network Challenges
Solving it with VXLAN EVPN
Hierarchical Topology No More Hair-Pining Control-Plane Learned

• Scale-Out • Default Gateway at every • Active Learning and
• Add more Spine for Leaf Distribution with BGP
bandwidth and EVPN
redundancy • Distributed Anycast
Gateway • Reduces the Broadcast
• Add more Leaf for port
capacity
Domain by configuring
• Flexible Network Service
VLANs where needed
• All Links are used (IP
Placement (L4-L7)
ECMP) • Pervasive Subnet and
Endpoint Mobility
First Design
Highly Available
Data Center
What defines a Highly Available Data Center?
Ask yourself
Do we have Network redundancy beyond a single rack? Do we have Application availability beyond a single rack?
Can we survive a single rack failure? How big is the Do we have Network and Application availability
change or failure domain? beyond a single location?
Is your DC Highly Available Yet?
Where do you stand? “The Single” Or…
Spine Spine Spine Spine

• Single Underlay Domain End-to-End
• Single Replication Domain for BUM

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
• Single Overlay Domain – End-to-End

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Encapsulation

• Single Overlay Control-Plane Domain – End-
to-End EVPN Updates
• Single VNI Admin Domain
“The Data Center of Yesterday”
Is your DC Highly Available Yet?
Where do you stand? “The Single” Or “The Multiple”
• Multiple Underlay Domains - Isolated

Spine
• Multiple Replication Domain for BUM – Interconnected
and Controlled
VTEP VTEP
• Multiple Overlay Domain – Interconnected and

Controlled
Spine Spine
• Multiple Overlay Control-Plane Domains –

VTEP VTEP
Interconnected and Controlled
VTEP VTEP
• Multiple VNI Admin Domain – Downstream VNI
“The Data Center of Today”
VXLAN EVPN Multi-Site
Architecture
VXLAN EVPN Multi-Site Overview
Functional Components
Region-A
Backbone
Site-External DCI Active Workloads
(Layer 2 & Layer 3)
Border Border
AZ-1 AZ-2
Gateway Gateway
Spine Spine
Site-Internal Fabric
Leaf VM VM Leaf
Application 1
Workload deployed across AZs
Network Fault Domain Network Fault Domain
Application Change Domain
Border Gateway
Deployment Considerations
Border Gateways main functions and use-cases:
• Packet Re-Origination (L2 and L3)
• Inter-Site DCI (East-West)
• L3 Extension (North-South) IP Handoff
• Connect L4-L7 services and EPs
• Integration with Legacy Networks (Co-existence and/or Migration) VXLAN EVPN

Greenfield
Classic Ethernet
Brownfield
• VXLAN L3-extension to the Public Cloud
Border Gateway
When to use what
Site 1 Site 2
Anycast Border Gateway VPC Border Gateway
• Up to 6 BGW • 2 BGW with Physical Peer-Link

• Simple Failure Scenarios • Small Deployments
• Any Deployments • End-Point or Network Services Connectivity on
BGW
• No End-Point or Network Services
Connectivity on BGW • Migration Use-Cases (Brownfield)
• Greenfield Deployments • Classic Ethernet / FabricPath to VXLAN EVPN
BGW Node Placement Option
Flexible Design option-1
Dedicated Border Gateway
Multi-Site DCI
Flexible scale-out approach for VXLAN EVPN Multi-Site DCI
Flexible Anycast or VPC BGW models
Capacity planning only for DCI traffic flows Border Gateway

V V
Clean role separation and uniform reachability from the entire
fabric are the major advantages
Border Gateway hosts: R R

VTEP for: Spine
- East-West (DCI Packet Re-Origination L2/L3)

V V V V
Leaf
V V = VTEP VXLAN EVPN Site
R = RR/RP
BGW on Spine nodes (Border Gateway Spine)
Flexible scale-out approach for VXLAN EVPN Multi-Site DCI Multi-Site DCI
Anycast BGW only
Extra functional dependency (BGW + Spine)

Border Gateway Spine
Capacity planning needs to accommodate all flows
V V
Border Gateway Spine hosts:
R R
VTEP for:

- Route Reflector (RR) (Site-Internal iBGP EVPN)
- Rendezvous Point (RP) (Site-Internal Multicast Underlay BUM) V V V V
Leaf
V V = VTEP VXLAN EVPN Site
R = RR/RP
BGW on top of Super-Spine nodes
Multi-Site DCI
Scale-out Multi-Clos Fabric to Interconnect the PODs using
Super-Spine
Architecture beyond a single server room. Simpler capacity Border Gateway

planning V V
Capacity planning only for DCI traffic flows
Clean role separation and uniform reachability from the entire R R

Super-Spine
Border Gateway hosts:
VTEP for: Spine

V V V V
Super-Spine hosts: Leaf
- Route Reflector (RR) (iBGP EVPN) POD-1 POD-2
- Rendezvous Point (RP) (Multicast Underlay BUM)
VXLAN EVPN Site
Border Node Placement Option
BGW on Super-Spine nodes (BGW Super-Spine)
Super-Spine Multi-Site DCI
Architecture beyond a single server room. Simpler capacity
planning
Capacity planning needs to accommodate all flows Border Gateway Super-Spine
Extra functional dependency (BGW + Super-Spine). Not V V

recommended due to Multi-POD failure dependency R R
Border Gateway Super-Spine hosts:

VTEP for:
Spine
- Route Reflector (RR) (Site-Internal iBGP EVPN) V V V V
- Rendezvous Point (RP) (Site-Internal Multicast Underlay BUM) Leaf
POD-1 POD-2
VXLAN EVPN Site
VXLAN Multi-Site Architecture
DCI- BGW to Cloud
• Scalable design option with horizontal scale-out
within and across multiple sites. Backbone Core
• Backbone/Cloud can be any routed service. (flat L3,

MPLS-L3VPN)
• Multi-Site Underlay: eBGP IPv4 Unicast

• Site-External DCI BUM: Ingress-Replication or
Multicast supported. *Currently NDFC supports only Border
Ingress-Replication. Gateway
• Site-Internal Fabric BUM: Ingress-Replication or

Multicast supported independently at each site.
Spine
• Multi-Site Overlay: eBGP EVPN Overlay
• Full-mesh BGP EVPN peering across all BGWs.
Leaf
• Ensure that PIP / VIP of all BGWs are known by
every BGW and MTU must accommodate VXLAN
AZ-1 AZ-2
encapsulated traffic.
Region-A
DCI- BGW Back-to-Back
• Available option for connecting 2 or 3 sites.
• Minimum topology is the square. An enhanced

option is to add links between site local BGWs for
improved ECMP and Failure scenarios.


Spine
Leaf
AZ-1 AZ-2
Region-A
Multi-Destination Traffic Forwarding
EVPN Route-Type 4 (Ethernet Segment Route)
• BGW advertises BGP EVPN Type-4 update to all of their BGP EVPN peers (Site-Internal and Site-External)
• BGW will use this information for DF (Designated Forwarded) election on per VLAN/VNI
• EVPN Type-4 update will also carry ES Import Route-Target Extended Community
• Remote site BGWs will not install the route due to different RT and Site ID
• Command to check: show bgp l2vpn evpn route-type 4
• BGP routing table entry for [4]:[0300.0000.00fd.e900.0309]:[32]:[10.1.1.1]/136
• [4] = EVPN Route-Type 4 evpn multisite border-gateway 65001
• [03]= ESI Type MAC-based delay-restore time 300
• [0000.0000.fde9] = ESI system MAC for Site-ID and RT {fde9 = 65001}
• [000309] = ESI local discriminator
• [32] = IP Address Length
• [10.1.1.1] = Sender (BGW Primary IP) used for DF election process
Border Border
Gateway Gateway
Site-ID: 65001 Site-ID: 65002
AZ-1 AZ-2
Region-A
BGW Designated Forwarded (DF) Election
• BGW learns about other BGWs originator IPs on the same site from exchanging EVPN Type 4 routes. The originator IP
address used is the BGW NVE source interface's primary IP address.
• Each BGW creates an ordinal list of originator IP in numerical order from lowest to highest. Every BGW is then given an
ordinal value based on its position in the ordinal list starting from position 0. The BGW with the lowest originator IP would
get an ordinal value of 0. The ordinal value decides which BGW will be the DF for a VLAN/VNI.
• DF election Formula uses “mod” operation:
• I = V mod N {I = ordinal value, V = VLAN #, N = # BGWs in a site}
• If there are 4 BGWs in one site. The ordinal list of the BGWs will be arranged as shown below.
• Ordinal List = 10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4 {I=0, I=1, I=2, I=3}
• The DF for VLAN 48 will be: “I = 48 mod 4 = 0 = 10.1.1.1”
EVPN Route-Type 3 (Inclusive Multicast Ethernet Tag Route)
• BGW advertises BGP EVPN Type-3 only towards Site-External DCI • Remote: Learning from Site 2 BGW to Site 1 BGW
• Receiving BGW switch does not forward it to Site-Internal Fabric BGP routing table entry for [3]:[0]:[32]:[20.3.0.4]/88
• Command to check: show bgp l2vpn evpn route-type 3, Path type: external, path is valid, is best path
• show l2route evpn imet all detail Imported to 1 destination(s)
• Local: Advertising from Site 1 BGW to Site 2 BGW Imported paths list: L2-10000
BGP routing table entry for [3]:[0]:[32]:[10.3.0.4]/88 AS-Path: 65002 , path sourced external to AS
AS-Path: NONE, path locally originated Extcommunity: RT:65001:10000 ENCAP:8
Extcommunity: RT:65001:10000 ENCAP:8 PMSI Tunnel Attribute:
PMSI Tunnel Attribute: flags: 0x00, Tunnel type: Ingress Replication
flags: 0x00, Tunnel type: Ingress Replication Label: 10000, Tunnel Id: 20.3.0.4
Label: 10000, Tunnel Id: 10.3.0.4
Path-id 1 advertised to peers: 20.2.0.4
Border Border
Gateway Gateway
Site-ID: 65001 Site-ID: 65002
AZ-1 AZ-2
Region-A
DCI- BGW Back-to-Back BUM (w/o Local L3 link)
DF VNI 4800
DF VNI 4800
Border Border
Gateway Gateway
AZ-1 AZ-2
Region-A
Drop due to Designated Forwarder (DF) rule
Physical connection
BUM packet
DCI- BGW Back-to-Back BUM (with Local L3 link)
DF VNI 4800
DF VNI 4800
Border Border
Gateway Gateway
AZ-1 AZ-2
Region-A
Drop due to Designated Forwarder (DF) rule
Physical connection
BUM packet
DCI- BGW Back-to-Back (with Local L3 link)
• Available option for connecting 2 or 3 sites.
• Minimum topology is the square. An enhanced

option is to add links between site local BGWs for
improved ECMP and Failure scenarios.


Spine
Leaf
AZ-1 AZ-2
Region-A
VXLAN Multi-Site Architecture Route Server
DCI- BGW to Centralized Router-Server

• BGP Route-Server (RS) function can be used in a
highly scalable design to act as the central EVPN
peering point for the Multi-Site. Backbone Core
• RS does not need to be on the data path.

• Site-External DCI BUM: Ingress-Replication or Multicast supported.
*Currently NDFC supports only Ingress-Replication.
• Site-Internal Fabric BUM: Ingress-Replication or Multicast supported Border
independently at each site. Gateway

• BGP EVPN peering only between BGWs and RS. Spine
• Route-Server (RS) must support…
• EVPN AFI and Router-Server function per RFC 7947
• Next-hop-unchanged, Retain RT, and RT Rewrite function Leaf
• Ensure that PIP / VIP of all BGWs are known by AZ-1 AZ-2
encapsulated traffic. Region-A
DCI- BGW to Centralized Router-Server Route Server
• BGP Route-Server (RS) function can be used in a
highly scalable design to act as the central EVPN
peering point for the Multi-Site.
• RS can also be on the data path.

• Site-External DCI BUM: Ingress-Replication or Multicast supported.
*Currently NDFC supports only Ingress-Replication. Border
• Site-Internal Fabric BUM: Ingress-Replication or Multicast supported Gateway
independently at each site.

Spine
• BGP EVPN peering only between BGWs and RS.
• Route-Server (RS) must support…

• EVPN AFI and Router-Server function per RFC 7947 Leaf
• Next-hop-unchanged, Retain RT, and RT Rewrite function
AZ-1 AZ-2
every BGW and MTU must accommodate VXLAN Region-A
Special Considerations for
Layer-3 extension across
Multi-Site
VXLAN Native Data Plane Security
Rogue VTEP
VTEP3
VXLAN Original • VXLAN EVPN offers a native data

VTEP3 VTEP2
Header Packet plane security functionality
• VXLAN traffic originated from a
VXLAN Original
VTEP1 VTEP2
Header Packet
remote VTEP is only accepted
when sourced from a TEP
EVPN Update
X address that is an “NVE peer”
VTEP1 VTEP2
• An NVE peer’s address is added
Traffic dropped as VTEP3 to the local table based on the
MP-BGP EVPN Table is not an NVE peer reception of MP-BGP EVPN
EP1 → VTEP1 updates carrying that specific
address as next-hop
EP1 NVE Peers EP2
VTEP1 • Prevents the insertion of rogue VTEPs
in a VXLAN EVPN fabric
Why can this become an issue for
Layer-3 communication across sites?
Inter-Site Layer 3 Traffic – Control Plane
Inter-Site Type-2 and Type-5 EVPN updates always carry the local Multi-Site VIP as next-hop address
• Only exception are Type-5 updates for L3 networks locally connected to the BGWs
Multi-Site VIP-1 EVPN Multi-Site Overlay Multi-Site VIP-2

10.10.0.1 10.10.0.2
PIP-1 PIP-2 PIP-11 PIP-22
10.13.0.1 10.13.0.2 MP-BGP EVPN 10.23.0.1 10.23.0.2
Border Border
Gateway Gateway
EVPN Update EVPN Update

MP-BGP EVPN Table MP-BGP EVPN Table
10.10.2.0/24 → 10.10.0.1 10.10.1.0/24 → 10.10.0.2
NVE Peers NVE Peers

10.10.0.1 10.10.0.2
EP1 EP2
10.10.1.1 10.10.2.1
Inter-Site Layer 3 Traffic – Data Plane
Inter-Site Layer 3 traffic is always sourced by local BGWs from their specific PIP address
Same applies to Intra-Site traffic between local BGW and leaf nodes
VXLAN Original
PIP-2 VIP-2
Header Packet
Multi-Site VIP-1 Multi-Site VIP-2

PIP-1
10.10.0.1
PIP-2
EVPN Multi-Site Overlay PIP-11
10.10.0.2
PIP-22
10.13.0.1 10.13.0.2 10.23.0.1 10.23.0.2
Border Border
Gateway NVE Peers Gateway
10.10.0.1
PIP-10 PIP-20
10.13.0.10 VXLAN Original VXLAN Original 10.23.0.10
PIP-10 VIP-1 PIP-11 PIP-20
Header Packet Header Packet
NVE Peers EP2

EP1 10.10.0.2
10.10.1.1 10.10.2.1
How to Prevent this issue?
Option-1: Advertise-PIP
Backbone • By default, all EVPN host routes (Route-Type 2) and

Layer-3 routes (Route-Type 5) are advertised with
the secondary IP address (VIP) of the vPC pair as the
VPC VIP VTEP and the BGP next-hop IP address for
10.13.0.3
PIP-1 V V PIP-2
reachability.
10.13.0.1 10.13.0.2
• While ARP/MAC/IPv6 ND entries are synced

between the peers of a vPC pair, prefix routes
R R belonging to an individual peer as well as external
routes received by a peer are not synced between
VPC peer switches. Using the VIP as the BGP next-
MP-BGP EVPN Table
V V
Backbone →
hop for these routes can cause traffic to be
10.13.0.3 forwarded to the wrong vPC peer and hence be
black-holed.
VXLAN EVPN Site
• While ARP/MAC/IPv6 ND entries are synced between the

Backbone peers of a vPC pair, prefix routes belonging to an individual
peer as well as external routes received by a peer are not
X synced between VPC peer switches. Using the VIP as the
VPC VIP BGP next-hop for these routes can cause traffic to be
10.13.0.3
V V forwarded to the wrong vPC peer and hence be black-holed.
PIP-1 PIP-2
10.13.0.1 10.13.0.2
• The Next-Hop on Leaf to send traffic towards Backbone is
still 10.13.0.3. Hence, traffic can still be hashed to Border-1.
R R At the same time, Border-1 lost its connection to Backbone.
Therefore, traffic will be black-holed.
• The Border-2 still has an active link towards the Backbone

MP-BGP EVPN Table and advertises the routes towards the Spine (RR). Later, the
V V
Backbone → Spine will reflect the route to Border-1, but it will reject it due
10.13.0.3
to Next-Hop being its own IP (VIP 10.13.0.3)
VXLAN EVPN Site
router bgp 65001
On both VPC peer Border
address-family l2vpn evpn
advertise-pip Devices. In case of VXLAN Multi-
Backbone Site, deploy the configs on local
Interface nve1 and remote BGW*
advertise virtual-rmac
* Maintenance window required.
VPC VIP
10.13.0.3
V V • The provision to use the primary IP address (PIP) on a vPC peer, as the
PIP-1 PIP-2
10.13.0.1 10.13.0.2 next-hop when advertising prefix routes (EVPN Route-type 5). This
includes direct, connected, attached, and external routes. In this way,
the traffic will always be destined to the right vPC peer destination
traffic will always then be advertised by the right vPC border peer that
R R was the BGP next-hop in the EVPN Route-type 5 advertisements.
• The advertise-pip command lets BGP use the PIP as next-hop when
MP-BGP EVPN Table advertising prefix routes or leaf-generated routes if vPC is enabled.
V V Backbone → With the advertise-pip and advertise virtual-rmac commands enabled,
10.13.0.1, 10.13.0.2 EVPN type 5 routes are advertised with PIP, and EVPN type 2 routes
are still advertised with VIP. In addition, a virtual MAC will be used with
the VIP that is shared by both vPC peers, and individual peer specific
VXLAN EVPN Site
system Router MAC will be used with PIP when the advertise-pip
feature is enabled.
Option-2: Per-VTEP, Per-VRF Loopback
• Define on all the BGW nodes a loopback interface (in a specific VRF)
and advertise the information across site with a Type-5 EVPN update
route-map fabric-rmap-redist-subnet permit 10

match tag 12345
!
interface loopback2
vrf member t1-vrf1
ip address 33.33.33.1/32 tag 12345
On local and remote BGW
!
router bgp 65001
vrf t1-vrf1
address-family ipv4 unicast
redistribute direct route-map fabric-rmap-redist-subnet
Option-3: Extend one L2VNI End-to-End
• An L2VNI must be stretched between the local leaf nodes and BGW nodes.
An L2VNI must be stretched across sites (i.e. defined on the local and remote
BGW nodes). Does not necessarily need to be the same L2VNI
vlan 99
vn-segment 9999
vlan 2300
name Net-App
vn-segment 30001
interface nve1
no shutdown On local and remote BGW
host-reachability protocol bgp
source-interface loopback1
multisite border-gateway interface loopback100
member vni 9999 associate-vrf
member vni 30001
multisite ingress-replication
mcast-group 239.1.1.1
Installing PIP Addresses as NVE Peers
Adopting any of the 3 solutions described in the previous slide ensures that the PIP addresses of
the BGWs can be installed as NVE peers in the local leaf nodes and in the remote BGW nodes
VXLAN Original
PIP-2 VIP-2
Header Packet
Multi-Site VIP-1 Multi-Site VIP-2

PIP-1
10.10.0.1
PIP-2
EVPN Multi-Site Overlay PIP-11
10.10.0.2
PIP-22
10.13.0.1 10.13.0.2 10.23.0.1 10.23.0.2
Border Border
Gateway NVE Peers Gateway
10.10.0.1, 10.13.0.1, 10.13.0.2
PIP-10 PIP-20
10.13.0.10 VXLAN Original VXLAN Original 10.23.0.10
PIP-10 VIP-1 PIP-11 PIP-20
Header Packet Header Packet
NVE Peers
EP1 10.10.0.2, 10.23.0.1, 10.23.0.2
EP2
10.10.1.1 10.10.2.1
Some more
Important Requirements!
Multi-Site Route-Target Problem
vrf context myvrf_50000 vrf context myvrf_50000
vni 50000 vni 50000
rd auto rd auto
address-family ipv4 unicast address-family ipv4 unicast
route-target both auto route-target both auto
route-target both auto evpn {65001:50000} route-target both auto evpn {65002:50000}
address-family ipv6 unicast address-family ipv6 unicast
route-target both auto evpn {65001:50000} route-target both auto evpn {65002:50000}
evpn evpn
vni 30000 l2 vni 30000 l2
rd auto rd auto
route-target import auto {65001:30000} route-target import auto {65002:30000}
route-target export auto {65001:30000} route-target export auto {65002:30000}
ASN ASN
EVPN Multi-Site Overlay
65001 65002
Border Gateway Border Gateway
VXLAN EVPN Site-1 VXLAN EVPN Site-2
EVPN Route-Target Re-Write
The ”rewrite-evpn-rt-asn” command modifies the incoming EVPN advertisements by swapping the
remote AS portion in the RT with the local ASN, provided the update is coming from a neighbor that is
locally configured.
router bgp 65001 router bgp 65002

neighbor 2.2.2.2 neighbor 1.1.1.1
remote-as 65002 remote-as 65001
update-source loopback0 update-source loopback0
ebgp-multihop 5 ebgp-multihop 5
peer-type fabric-external peer-type fabric-external
address-family l2vpn evpn address-family l2vpn evpn
send-community send-community
send-community extended send-community extended
rewrite-evpn-rt-asn rewrite-evpn-rt-asn
ASN ASN
65001 65002
Route-Server in DCI
EVPN Route-Target Re-Write by Route-Server
router bgp 65099
retain route-target all
template peer OVERLAY-PEERING
update-source loopback0
ebgp-multihop 5
send-community
send-community extended
route-map unchanged out
neighbor 20.2.0.4
inherit peer OVERLAY-PEERING
remote-as 65002
address-family l2vpn evpn Route-Server
rewrite-evpn-rt-asn
ASN ASN
65001 ASN 65099 65002
Peer-Type Fabric-External
EVPN Split-Horizon and Route Re-Origination
• The command is defined towards Site-External EVPN peering.

It provides the capability of VXLAN packet re-origination and
router bgp 65001 implements VPN split horizon mechanism.
neighbor 2.2.2.2
remote-as 65002
• EVPN route coming from Site-External peer-type must not be
update-source loopback0
ebgp-multihop 5 re-advertised back into the VPN. The route is only advertised
peer-type fabric-external towards Site-Internal VTEPs
send-community
send-community extended
rewrite-evpn-rt-asn
ASN ASN
65001 65002
Failure Detection on BGWs
Use of Interface Tracking
• Because of the critical role played by the BGWs, it is
critical to consider the required behavior during different
DCI failure scenarios.
Site-External DCI • Tracking the state of the interfaces connecting the BGWs
to the spines (“fabric-tracking”) and to the ISN (“dci-
tracking”)
Border
Gateway
• The “dci-tracking” configuration is also required on
Layer-3 interfaces locally connecting Anycast BGW
nodes, needed for example, back-to-back topologies.
Spine • Allows to define via configuration how the BGW is
connected in the topology and to take proper action
depending on the specific failure.
Leaf
• The BGW node that gets isolated from the fabric or from
evpn multi-site dci-tracking the ISN needs to stop receiving traffic flows if it can not
forward them to the destination.
evpn multi-site fabric-tracking
Fabric Isolation
• The Site-Internal interfaces on BGW nodes are constantly
tracked to determine their status (‘evpn multisite fabric-
DCI tracking’ command)
Site-External DCI
Multi-Site VIP
10.100.1.1
Border
Gateway
PIP PIP PIP PIP
10.10.10.1 10.10.10.2 10.10.10.3 10.10.10.4
Spine
Leaf
Fabric Isolation • The Site-Internal interfaces on BGW nodes are constantly
tracked to determine their status (‘evpn multisite fabric-
tracking’ command)
DCI
• If all the Site-Internal interfaces are detected as down on a
BGW:
Site-External DCI
1) The BGW will isolate itself from Multi-Site traffic. Hence , it will
Multi-Site VIP shutdown Lo100 aka VIP and stop advertising to remote sites. So,
10.100.1.1
the traffic from remote site will never come to this BGW as it will no
Border
longer be part of the ECMP.
Gateway PIP PIP PIP
10.10.10.2 10.10.10.3 10.10.10.4 2) The BGW site external BGP session will be up and running but it
will withdraw all EVPN routes (Type 2,3,4,5)
3) Next, the remaining BGWs will receive that type 4 withdrawal from
the failed BGW, and it will re-run the DF election. For example , if
Site-Internal Fabric BGW 1 was DF for VLAN 10 then a new election will happen
Spine
between the remaining BGWs.
• As a result, the BGW becomes isolated from both the Site-

Leaf Internal and Site-External networks.
• Seamless BGW node re-insertion using a “delay-restore” timer

for the Multi-Site VIP address
DCI Isolation
• The Site-External interfaces on BGW nodes are
constantly tracked to determine their status (‘evpn
DCI multisite dci-tracking’ command)
Site-External DCI
Multi-Site VIP
10.100.1.1
Border
Gateway
PIP PIP PIP PIP
10.10.10.1 10.10.10.2 10.10.10.3 10.10.10.4
Spine
Leaf
DCI Isolation • The Site-External interfaces on BGW nodes are constantly
tracked to determine their status (‘evpn multisite dci-tracking’
command)
DCI • If all the Site-External interfaces are detected as down on a

BGW:
Site-External DCI
1) The BGW will stop advertising Multi-Site VIP aka Lo100 towards
Multi-Site VIP Site-Internal network. Hence, all traffic destined to remote sites will
10.100.1.1 be re-routed to remaining BGWs due to ECMP.
Border
Gateway 2) The isolated BGW will also withdraw EVPN Type-4 as new DF
PIP PIP
PIP
10.10.10.1
PIP
10.10.10.3 10.10.10.4
election is triggered amongst the remaining BGWs.
10.10.10.2
3) The isolated BGW do not need to withdraw Type-2,3, and 5 routes

as DCI interfaces and remote BGP EVPN peering are down.
Site-Internal Fabric • As a result, the BGW continues to operate as a Site-Internal

Spine
VTEP as it’s PIP remains up. Hence, if any external network is
connected to this BGW, traffic from local site can still be sent
to this BGW.
Leaf
• Seamless BGW node reinsertion using a “delay-restore” timer
for the Multi-Site VIP address
Cool! But What’s the Catch?
There is always a catch ☺
1 This seems really cool! Is it easy to configure?

There are lots of moving parts: OSPF/BGP/VXLAN. Manual
configuration can be challenging.
2 How easy is it to make changes?

You still rely on traditional SSH based management to each device,
which can be cumbersome and error prone.
3 How much Visibility do I have into the network?

Visibility and Troubleshooting is still performed on a “switch-by-
switch” basis.
The DC Needs to Go Anywhere the Data Is
IoT Edge
Enterprise DC 5G Telco Edge
Public Cloud/IaaS Enterprise Edge
Private Cloud Colo / Bare Metal Cloud
And your world is constantly changing…
Campus Branch Data Center Edge & IoT
What is Nexus
Dashboard
Fabric
Controller?
Cisco Nexus Dashboard Powering automation
Unified agile platform
Simple to automate, simple to consume
Insights
Orchestrator Fabric Controller
Data Broker SAN Controller

Consume all services in one place
Private cloud Third-party Apps Public cloud
Cisco Nexus Dashboard Fabric Controller
App accessed through Cisco Nexus Dashboard
Cisco Nexus Dashboard Access NDFC
Benefits Automation Management and Compliance Visibility and Monitoring
Cisco Nexus Dashboard Fabric Controller
Automation Management Visibility

Accelerate provisioning In depth Management Get Centralized Visibility
and simplify deployments and control for all and Monitoring views
network deployments
Benefits of NDFC 12 over DCNM 11
Plan your migration today!
Simplified Installation from Cisco App Store
Single Experience with a common Web GUI which simplifies adoption across
the entire Data Center Networking Product Portfolio
Easier implementation of various personas namely LAN, SAN, IPFM controller
Easier scalability with adding extra nodes to the cluster dynamically
Run migration
DCNM to NDFC Migration script
Y DCNM IPs N
re-used for
ND nodes
Shutdown Deploy ND
DCNM cluster
Deploy ND Add External

Persistent
cluster
IPs
Cisco Data Center Cisco Nexus

Add External
Dashboard Persistent Install
Network Manager IPs NDFC App
(DCNM) Fabric Controller
(NDFC) Install Restore
NDFC App Backup file
Restore Shutdown
Backup file DCNM
Automation
Accelerate provisioning from days to minutes
Easy to understand approach to

auto-bootstrapping of entire fabric
Rapid Deployment with Fabric Builder

best practice templates for VXLAN-EVPN
Optimized for both large deployments

and traditional deployment models
Service Insertion and Layer-3 handoff
DevOps friendly
Benefits
Simplify fabric deployments Developer agility VXLAN EVPN Multi-Site
Management Single point for management
for data center operations
Optimized for both large deployments
and traditional deployment models
Granular RBAC
Image management
RMA
Scale within and across data centers

with Nexus Dashboard Orchestrator
Management for non-Nexus platforms
Benefits
Reliability Compliance Secure
Visibility &
Monitoring Get comprehensive monitoring
Enhanced topology views
Compute and endpoint visibility
VXLAN OAM support with NDFC
Obtain detailed inventory, health, resource

consumption information on devices
End-to-end visibility, monitoring

and troubleshooting
Integrate with NDI for Day 2 operations
Benefits
Intuitive Deep visibility Enhanced monitoring
Cisco NDFC Modes
Make decision at run-time!
Runtime Feature Installer Easy switch between modes
Fabric discovery for SAN controller for MDS Fibre

LAN Deployments Channel Deployments
Fabric controller for LAN

and IPFM Deployments
Fabric Discovery
Run fabric discovery for LAN deployments:

Enable inventory, discovery, monitoring only
Enable Cisco Nexus Dashboard’s Day 2 operations

capabilities without deploying fabric controller
You can switch anytime from Fabric Discovery to

Fabric Controller Mode
Benefit
Deep visibility into deployments 74

BRKDCN-2988 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Controller
Provides fabric management for multiple types of

LAN solutions, including VXLAN-EVPN, and
traditional 3-tier LAN deployments
Compliance management ensures that network is in

sync with intended deployments and allows users to
deploy any corrections
Benefit
Most configurations are automatically done following Cisco Best Practices 75

SAN Controller
Completely redesigned web-based zoning interface

to drastically reduce the cycle time for common
administration tasks. Provides IVR zoning function as
well, all on the same page.
SAN Insights provides useful data to the

administrators so they can be fully aware about the
fabric status
Benefit
Transition to a web-based configuration method is made easy 76

Cisco Nexus Dashboard
Physical ND Cluster
Orchestrator Insights Fabric Controller

VM VM VM
VM VM
Data Broker SAN Controller
Virtual ND Cluster
Cisco Nexus Dashboard Formats - NDFC
Physical ND Cluster Virtual ND Cluster
Each node is a UCS Server with: For NDFC each vND VM must
satisfy the following requirements:
2.8GHz AMD CPU Specs APP
256G RAM
4x2.4TB HDD vCPU 16
960 GB SSD
1.6 TB NVMe drive RAM (GB) 64
For the latest information check the specific SSD (GB) 550
scalability guide. 12.1.3b Verified Scalability
Cisco Nexus Dashboard Scaling - NDFC Cisco NDFC 12.1(3)
Physical ND Virtual ND VXLAN EVPN VXLAN EVPN

Cluster Cluster (Greenfield) (Brownfield)
Full scale for NDFC can Full scale for NDFC can Switches per Fabric: Switches per Fabric:
be achieved with 3 be achieved with 5 200 200
nodes nodes
Overlays: 500 VRF and Overlays: 400 VRF and
Managed mode (VXLAN Managed mode (VXLAN 2000 Layer-3 Networks 1050 Layer-3/Layer-2
and BGP fabrics): 500 and BGP fabrics): 400 OR 2500 Layer-2 Networks
switches switches Networks
Multi-Site Domain: 30
Managed/Monitor mode Managed/Monitor mode Multi-Site Domain: 30 fabrics
(External fabrics): 1000 (External fabrics): 1000 fabrics
switches switches
ToR/Leaf: 40 Leaf
Overall fabric count: 50 Overall fabric count: 50 (VTEP) and 320 ToRs in
DC VXLAN EVPN fabric
Recommended 3x vND can support 100
switches in managed
mode
In any case at least 3 nodes

must be deployed for proper
redundancy. 1x vND also
supported for Production
Nexus Dashboard
Interface Types
• Each ND node has two interface types:

ND Mgmt. Interface
• Management Interface: should be dedicated to the management of

the ND cluster → connectivity to NTP and DC Proxy servers,
Intersight, DNS, ND (and ND Apps) UI access and to perform
ND Fabric Interface firmware upgrade (for ND or Apps)
• Fabric Interface: used for the bring up of the ND cluster (node to

node communication) and application to application (NDO, NDI,
NDFC, etc.) communication
Cisco Nexus Dashboard Connectivity - NDFC
Data/Fabric Network The two interfaces cannot share the
L2
bond0br same subnet
MAX RTT 50msec Intra/Inter APP PTP

ND Clustering
10.1.1.101/24 10.1.1.102/24 10.1.1.103/24 Switch Access*
DNS SNMP TRAPS
POAP DHCP
10.2.2.101/24 NTP
GUI Access
10.2.2.102/24 10.2.2.103/24
10.2.2.150/24 10.2.2.151/24
MAX RTT 50msec CLI via SSH

DC App Center
Management Network Intersight
bond1br * by default
NDFC Persistent IPs
Persistent IPs are tied to a
service, like the SNMP trap
receiver
If the SNMP trap POD gets

re-spawned into a different
ND host the sticky IP will be
moved there
L2 adjacency uses ARP, L3

adjacency BGP
announcements
NDFC Persistent IPs – Normal conditions
Data/Fabric Network
bond0br
10.1.1.101/24 10.1.1.102/24 10.1.1.103/24
10.2.2.101/24 10.2.2.102/24 10.2.2.103/24

10.2.2.150/24 10.2.2.151/24
Management Network
bond1br
NDFC Persistent IPs - Failover
Data/Fabric Network
bond0br
10.1.1.101/24 10.1.1.102/24 10.1.1.103/24
10.2.2.101/24 10.2.2.102/24 10.2.2.103/24

10.2.2.150/24 10.2.2.151/24
Management Network
bond1br
Data/Fabric Network bond0br
L3
This L3 options is valid since
MAX RTT 50msec
12.1.1e
172.17.20.127/24 172.17.20.128/24 Each ND node on a different

10.1.1.101/24 10.1.2.101/24 10.1.3.101/24
Subnet
BGP Sessions are

10.2.1.101/24 10.2.2.102/24 10.2.3.103/24 established for Persistent IP
advertisement (no multi-hop)
MAX RTT 50msec
Persistent IPs must not
overlap with ND subnets
Management Network bond1br
L3
MAX RTT 50msec
172.17.20.127/24 172.17.20.128/24
10.1.1.101/24 10.1.2.101/24 10.1.3.101/24
10.2.1.101/24 10.2.2.102/24 10.2.3.103/24
MAX RTT 50msec

L3
MAX RTT 50msec

172.17.20.127/24
172.17.20.128/24
10.1.1.101/24 10.1.2.101/24 10.1.3.101/24
10.2.1.101/24 10.2.2.102/24 10.2.3.103/24
MAX RTT 50msec

Cisco NDFC Connectivity to the Switches Use case #1
Discovery and Deployment happen

via ND Management Interface as that
subnet is directly connected
Fabric/Inband
bond0br ND Data Interface eventually used for
ND Cluster Endpoint Locator Feature (BGP
bond1br
SW1 SW2 SW3 towards Spine RR)
OOB mgmt0 mgmt0 mgmt0
Persistent IPs are allocated on the

MAX RTT
Management Subnet
50msec
ND Mgmt ND Data Subnet Fabric Mgmt 0 Fabric Inband

Subnet Subnet Subnet Works by default!
10.2.2.0/24 10.1.1.0/24 10.2.2.0/24 10.3.3.0/24
Everything is done over the ND Data

Interface as that subnet is directly
connected
bond0br
Fabric/Inband
ND Cluster Data Subnet
bond1br
SW1 SW2 SW3
MAX RTT
50msec
ND Mgmt ND Data Subnet Fabric Mgmt 0 Fabric Inband LAN Device Management
Subnet Subnet Subnet Connectivity must be set to
Data
10.2.2.0/24 10.1.1.0/24 10.1.1.0/24 10.3.3.0/24
(see next slide)
continues
The change is global for the

NDFC Instance
Persistent IPs will be

provisioned over ND Data
Interface
Settings --> Server Settings

--> LAN Device
Management Connectivity
Discovery and Deployment happen

via ND Management Interface
ND Data Interface eventually used for

Fabric/Inband Endpoint Locator Feature (BGP
bond0br
towards Spine RR)
ND Cluster
bond1br
SW1 SW2 SW3
Management Subnet
MAX RTT
50msec
ND Mgmt ND Data Subnet Fabric Mgmt 0 Fabric Inband A static route to 10.4.4.0/24
Subnet Subnet Subnet must be added in ND
Management Interface
10.2.2.0/24 10.1.1.0/24 10.4.4.0/24 10.3.3.0/24
(see next slide)
continues
The static route needs to be

added in the Nexus
Dashboard Control Panel.
Infrastructure --> Cluster

Configuration --> Routes
Everything is done over the ND Data

Interface as that subnet is directly
connected
Fabric/Inband
bond0br Persistent IPs are allocated on the
ND Cluster Data Subnet
bond1br
SW1 SW2 SW3
LAN Device Management

MAX RTT Connectivity must be set to
50msec Data
ND Mgmt ND Inband Fabric Mgmt 0 Fabric Inband
Subnet Subnet Subnet Subnet A static route to 10.4.4.0/24
must be added in ND Data
10.2.2.0/24 10.1.1.0/24 10.4.4.0/24 10.3.3.0/24
Interface, not for routing but
for POAP
Why do YOU
need NDFC?
BRKDCN-2988 94
Why NDFC?
Multi-Architecture
Layer-3 MSDC Fabric
Campus VXLAN EVPN
Data Center VXLAN EVPN

VXLAN EVPN Multisite
Traditional 3-Tier LAN
Why NDFC?
Multi-Topology, Multi-Protocol
Rich set of control plane and data plane possibilities available
VXLAN BGP EVPN

vPC BGP Routed
MPLS FHRP
STP
VLAN IS-IS
FabricPath OSPF
IP
IPFM
Why NDFC?
Multi-Domain, Multi-Platform
NX-OS Nexus 9000 and 3000 IOS-XE Catalyst 9000 IOS-XR ASR 9000
Supported Hardware and Software

might vary depending on NDFC version
Check compatibility matrix 12.1.3b
NX-OS Nexus 7000 IOS-XE ASR 1000
Why NDFC?
In a nutshell…
Step into SDN via VXLAN BGP EVPN Multi-OS management and support
Config and Compliance across Cisco Products Simplify Complex Network Operations
Single Source of Truth Automate, Manage, and Interconnect

Multi-Fabric topologies
End to End Automation Layer-3 Boundary across Zones, L2/L3

across IOS-XE, NXOS, and Multicast Overlay
Single Pane of Glass for Day-0/Day-1 Provisioning Programmability and Orchestration
Automate VXLAN EVPN deployments
Provision a new fabric in minutes
Un-provisioned switches Cisco best practice implemented
VXLAN Fabric
Support for both brownfield and Fast, automated process

greenfield deployments
Benefit
Accelerate fabric deployments Automated consistency Minimize risk Support for both Greenfield and Brownfield deployment
Manage and
Deploy
VXLAN EVPN
Multi-Site with
NDFC
VXLAN EVPN Multi-Site deployment with NDFC
Deploy in a few minutes!
Deploy your individual DC fabrics (Data Center VXLAN EVPN)
(Optional) Deploy a fabric for Route-Server (Multi-Site Interconnect Network)
Deploy a fabric for the Multi-Site (VXLAN EVPN Multi-Site) & import all the
previous fabrics into this one
Recalculate and deploy in the VXLAN EVPN Multi-Site Fabric
VXLAN BGP EVPN Greenfield
Not on VXLAN EVPN Today?
NDFC Fabric Controller Build VXLAN fabric Templates already IP addresses, overlay pool,
Mode in few minutes embed best practices routing profiles, replication
attributes –all taken
care by NDFC
Cisco best practice

Step 2 implemented
Discover
Import switches with POAP or Day-0 config
Define switch Roles (Border, Leaf, Spine, etc)
[Optional] Create vPC pairs VXLAN Fabric
Step 1 Step 3
Create Recalculate and Deploy Fast, automated process
Define fabric settings (Underlay, Overlay) - Generates config based on intent
AS#, Replication Mode, IGP, IP Pools, etc. Preview side by side diffs
Step 1 -> Create a Fabric
Create Fabric
(continued)
BGP ASN
VXLANv4
or
VXLANv6
Underlay IP > /30 or /31
IGP > OSPF or ISIS
RR > 2 or 4
Distributed Anycast Gateway
PM Metrics
(continued)
BUM >
Multicast or
Ingress
Replication
L2VNI
Multicast
Group
RP > 2 or 4
(continued)
Cisco’s Best
Practice
Configuration
Templates
VXLAN Overlay
Mode > CLI or
Config-Profile
(continued)
L2VNI Label
L3VNI Label
Router ID
VRF_LITE
VTEP IP Handoff
RP IP
P2P Underlay IP
VRF_LITE IP
Range
L4-L7 Service
Network
(continued)
NDFC Built-In
Bootstrap POAP
Services. Supports
OOB and In band
POAP
Step 2 -> Add Switches
Add Switches
(continued)
Switch Mgmt0 IP
Switch Discovery
Credentials
Switch Hops
based on CDP
Discover
Switches
VXLAN Greenfield or Brownfield
(continued)
Switch Inventory
Management
Add Switches
Step 3 -> Set Role
Topology View
Switch Roles
Step 4 -> VPC Pairing
(optional)
Hierarchical Topology
Leaf VPC Pairing
Step 4 -> VPC Pairing
(optional)
Step 5 -> Recalculate and Deploy
Fabric
Recalculate and
Deploy
Pending
Configuration
(continued)
Fabric
Configuration
status
Pending
Configuration
(continued)
Spine related
features
CDP Link &

Spine RR/RP IGP configs
function
Spine EVPN
RR Client
(continued)
VPC Best Practice

Configs
Switch running config

v NDFC Intent
NDFC VXLAN EVPN Topology View
Link Status
Fabric Fabric
Operational Configuration
View View
Fabric
Configuration
Status
Fabric Minor
Alarms
Step 1 & 2 -> Create Individual Fabrics
Set appropriate roles
BGP EVPN
DC1-VXLAN ROUTER-SERVER DC2-VXLAN
BORDER
GATEWAY-
BORDER SPINE
GATEWAY BACKBONE-WAN
SPINE
LEAF
LEAF
Step 3 -> Create VXLAN EVPN Multi-Site Fabric
Create Fabric
(continued)
L2VNI Label
(for stretched
overlays)
Distributed Anycast Gateway
(for EP Mobility)
L3VNI Label
(for stretched
overlays) Multi-Site BGP NH for
VXLAN EVPN on BGWs
(continued)
VXLAN Multi-Site
BGW Design Model
BGP EVPN Router-

Server Loopback IP
for EVPN peering
BGP ASN of Route-Server
Auto Deploy Multi-Site

Underlay Configs
Enable BGP send-community
(continued)
Multi-Site Loopback IP
range allocation on BGWs
DCI Underlay P2P Subnet
DCI Underlay P2P Subnet

Mask
(continued)
Add Child (VXLAN and

BACKBONE WAN) fabrics inside
VXLAN Multi-Site container
Individually Add Fabrics Once Child fabrics are

added, perform
Recalculate and Deploy
Step 4 -> Recalculate & Deploy
In VXLAN EVPN Multi-Site Fabric
EVPN
Overlay
Feature eBGP IPv4 Unicast
(default) DCI Underlay
DCI Underlay peering towards BGW
Routed Port
towards BGW
Retain RT coming eBGP EVPN DCI Overlay
from BGW EVPN NLRI peering towards BGW
EVPN NH unchanged on RS for

BGW-to-BGW VTEP termination
Rewrite ASN
portion of Auto RT
Step 4 -> Recalculate & Deploy
BGW VTEP Interface

Lo1 = NVE PIP
BGW Interface Lo100 = NVE Multi-Site
advertisement for Anycast VIP
DCI reachability
BGW Site ID
eBGP EVPN DCI Overlay
peering towards RS
BGW DCI-tracking towards

Site-External underlay
Rewrite and Re-origination

BGW Fabric-tracking towards
function on BGW
Site-Internal underlay
eBGP IPv4 Unicast

(default) DCI
Underlay peering Rewrite ASN
towards RS portion of Auto RT
NDFC VXLAN EVPN Multi-Site Topology View
Site-External DCI
Deploy VXLAN EVPN Overlays
in Multi-Site Architecture
Have VXLAN EVPN Multi-Site ready
Create Network and VRF from Multi-Site Fabric
Attach Network and VRF to switches

in Child fabrics
(optional) Preview the attached configuration
Deploy the configuration
Step 1 -> Navigate to VXLAN Multi-Site Fabric
Select Multi-Site Fabric
*UI Navigation: LAN Fabrics > Select Fabric > Fabric Overview > Networks > Create
Step 2 -> Create VRF and Network
Auto
Generated
Name
L2 or L3
Network
VXLAN
Overlay VRF
Auto
Generated
L2VNI Label
Network SVI
Step 3 -> Attach VRF and Network
Select Network
Selective
Edit for selecting
Network
port attachments
Attachments One-Click attachment
for all fabrics
Step 3 -> Attach VRF and Network
(continued)
Select ports for the

network
attachments
Step 4 -> Preview and Deploy VRF and Network
Deploy pending
network
attachment config
VRF and Network Configs
L3VNI VRF
Configs
VXLAN Tenant
L2VNI
Network
Configs
NVE Tunnel
Configs
VRF and Network Configs
L3VNI VRF
Configs
L3 SVI
VXLAN Tenant Configs
L2VNI
Network
Configs
NVE Tunnel
Configs
VRF and Network Deployment Status
VXLAN Overlay
Deployment Status
VXLAN EVPN Brownfield Migration
Already using VXLAN?
Want to use NDFC for managing your fabric?
NDFC fully supports Non-disruptive import Learns topology, all Start Managing fabric
Brownfield of existing VXLAN configuration, as if provisioned from
EVPN deployments associated resources, NDFC
IP subnets, VNIs,
VLANs, etc.
Step 2
Discover
Import switches with Preserve Config
Define switch Roles (Border, Leaf, Spine, etc)
Step 1 Step 3
Create Recalculate and Deploy
Define fabric settings (Underlay, Overlay) -
Sanity checks for mis-config and Normalizes configuration
Match AS#, Replication Mode, IGP, etc.
to best practices
Manage and Deploy
External IP Handoff
with NDFC
External Handoff Placement
Border Role compared to Leaf Role
• Leaf VTEP (East-West traffic) External Network
• Connectivity to endpoints, the first-hop routing

• Server-to-server traffic Edge Router
• Border Leaf VTEP (North-South traffic) IPv4/IPv6/L3VPN

• Transit node
V V Handoff
• Traffic destined from within the VXLAN EVPN Border Leaf
North-South
Fabric to outside or vice versa L2VPN EVPN
• Optionally it can have directly attached endpoints R R
Spine
• Separating Roles simplifies operations
V V V V
Leaf
V V = VTEP
VXLAN EVPN
R = RR/RP East-West
Border Node Design
VXLAN EVPN Fabric
External Network
Border as Leaf nodes (Border Leaf)
Capacity planning only for north-south traffic flows Edge Router

Clean role separation and uniform reachability from the
entire fabric are the major advantages IPv4/IPv6/L3VPN
Support for Inter-AS option A (VRF-LITE) and seamless V V Handoff
VXLAN-MPLS gateway (Border-PE) Border Leaf
North-South
Border Leaf hosts: L2VPN EVPN
R R
VTEP Mainly for Spine
• North-South V V V V
Leaf
VXLAN EVPN
East-West
Border as Spine nodes (Border Spine) External Network
Leaf endpoints are one-hop away from External Networks
Allowance for a scale-out approach to add spines and provide Edge Router
more bandwidth and resiliency
Support for Inter-AS option A (VRF-LITE) and seamless IPv4/IPv6/L3VPN

VXLAN-MPLS gateway (Border-PE)
V R V R Handoff
Extra functional dependency (Border + Spine) Border Spine
North-South
L2VPN EVPN
Border Spine hosts:

VTEP for:
• North-South V V V V
• East-West Leaf
• Route Reflector (RR) (iBGP EVPN) VXLAN EVPN

• Rendezvous Point (RP) (Multicast Underlay BUM)
East-West
Border on top of Super-Spine nodes (Border Leaf) External Network
Super-Spine
Edge Router
planning IPv4/IPv6/L3VPN
V V
Capacity planning only for north-south traffic flows Border Leaf Handoff
North-South
Clean role separation and uniform reachability from the entire R R L2VPN EVPN
Super-Spine
Support for Inter-AS option A (VRF-LITE) and seamless Spine

VXLAN-MPLS gateway (Border-PE)
V V V V V V V V
Border Leaf hosts: Leaf
VTEP Mainly for
• North-South POD-1 POD-2
Super-Spine hosts:
• Route Reflector (RR) (iBGP EVPN) VXLAN EVPN Site-1
East-West
Border as Super-Spine nodes (Border Super-Spine)
External Network
Super-Spine
Edge Router
planning
IPv4/IPv6/L3VPN
Support for Inter-AS option A (VRF-LITE) and seamless V R V R
VXLAN-MPLS gateway (Border-PE) Border Super-Spine Handoff
North-South
L2VPN EVPN
Extra functional dependency (Border + Spine). Not Spine

recommended due to Multi-POD failure dependency
Border Super-Spine hosts: V V V V V V V V

VTEP Mainly for Leaf
• North-South
• East-West POD-1 POD-2
• Route Reflector (RR) (iBGP EVPN)
VXLAN EVPN Site-1
East-West
Border Node
Design
VXLAN EVPN
Multi-Site
Flexible Design option-1 External Network
Border as BGW nodes (Border Gateway)

Edge Router
Flexible scale-out approach with L3 handoff alongside VXLAN
Multi-Site DCI
IPv4/IPv6/L3VPN
Alternative option to the use of dedicated Border Leaf nodes
Multi-Site DCI
BGW-to-BGW L3 VXLAN DCI or BGW-to-External L3 IP handoff
Flexible Anycast or VPC BGW models L2VPN EVPN
North-South
Support for Inter-AS option A (VRF-LITE) Border Gateway Border Gateway
V V V V
Seamless VXLAN-MPLS gateway (Border-PE) not supported
Capacity planning needs to accommodate all flows R R R R Spine

Spine
Chances of traffic hair-pinning unless Type-2 host routes V V V V V V V V

advertised Leaf Leaf
Border Gateway hosts: VXLAN EVPN Site-1 VXLAN EVPN Site-2
VTEP for:
• North-South (L3 Handoff)
• East-West (DCI Packet Re-Origination L2/L3) East-West
External Network
Edge Router
IPv4/IPv6/L3VPN
Border as Shared Border nodes (Border Leaf)
V V
Flexible scale-out approach with separate External IP handoff and DCI Shared Border
connectivity
L2VPN EVPN
Shared Border acts as a regular Border Leaf in Site-External fabric
Independent of VXLAN Multi-Site software, hardware, and licensing

dependencies Multi-Site DCI
North-South
Deterministic traffic flows and failure scenarios (N-S and E-W)
L2VPN EVPN
Support for Inter-AS option A (VRF-LITE) and seamless VXLAN-MPLS Border Gateway
Border Gateway
gateway (Border-PE)
V V V V
Capacity planning only for north-south traffic flows
Easily connect shared services (such as L4-L7) for symmetric Layer- R R R R Spine
Spine
2/Layer-3 traffic flows
Optimized traffic flows and reduces natural traffic hair-pinning V V V V V V V V

Leaf Leaf
Shared Border Leaf hosts:
VTEP Mainly for
• North-South
East-West
External Network
Border as dedicated Leaf nodes (Border Leaf)
Edge Router
Clear separation between Border Leaf and BGW
roles
IPv4/IPv6/L3VPN
Dedicated bandwidth and resiliency for External
and DCI traffic Multi-Site DCI
Capacity planning based on traffic type (External v

L2VPN EVPN
North-South
DCI)
Border Leaf Border Gateway Border Gateway Border Leaf
Support for Inter-AS option A (VRF-LITE) and
seamless VXLAN-MPLS gateway (Border-PE) V V V V V V V V
Additional CAPEX and OPEX costs R R R R

Spine Spine
Border Gateway hosts:
VTEP for: V V V V V V V V
• East-West (DCI Packet Re-Origination L2/L3) Leaf Leaf

Border Leaf hosts:
VTEP for:
• North-South East-West
Flexible Design option-4 External Network
Border as BGW-Spine nodes (Border Gateway Spine)

Edge Router
Flexible scale-out approach with L3 handoff alongside VXLAN
Multi-Site DCI
IPv4/IPv6/L3VPN
Alternative option to the use of dedicated Border Leaf nodes
BGW-to-BGW L3 VXLAN DCI or BGW-to-External L3 IP handoff Multi-Site DCI
Support for Inter-AS option A (VRF-LITE) L2VPN EVPN
North-South
Extra functional dependency (BGW + Spine) Border Gateway Spine Border Gateway Spine
V R V R V R V R
Seamless VXLAN-MPLS gateway (Border-PE) not supported

Border Gateway Spine hosts:
VTEP for: V V V V V V V V
• North-South (L3 Handoff) Leaf Leaf
• East-West (DCI Packet Re-Origination L2/L3)
• Route Reflector (RR) (iBGP EVPN)
• Rendezvous Point (RP) (Multicast Underlay BUM) East-West
Who can be the
External Edge
Router?
Edge Router Placement Option
Supported Hardware and Software
might vary depending on NDFC version
Check compatibility matrix 12.1.3b
Nexus and Non-Nexus support

External Network External Network External Network
Edge Router Edge Router Edge Router
Border Leaf Border Leaf Border Leaf
Data Center VXLAN EVPN Data Center VXLAN EVPN Data Center VXLAN EVPN
Managed IOS-XR as Edge Router Managed IOS-XE as Edge Router

Managed NX-OS as Edge Router
- Nexus 9000 - Cisco 8000 - Catalyst 9000
- Nexus 7000 - ASR 9000 - Catalyst 8000
- NCS 5500 - ASR 1000
- CSR 1000
Border to External
Peering Type
External Layer-3 connectivity options
Inter-AS Option A
Clear separation of Autonomous Systems Separated Border + PE
Simple, Straight forward, and Commonly used (Inter-AS Option A)
No need for redistribution VPNv4/VPNv6 External Network
MPLS-LDP or SR-MPLS
Easy and Flexible BGP route-filtering mechanisms
Edge Router BGP 65099
BGP natural loop avoidance
Structured handoff between the VXLAN BGP EVPN fabric and the IPv4/IPv6
external routing domain (Backbone, WAN, Campus, etc.)
V V
Not ideal for High scale VRF handoff deployment Border Leaf BGP 65001
Peering Type = Sub-interfaces on physical routed (or L3 Port-

channel) interfaces R R
Spine
• Sub-interface with dot1q tag to mark the traffic to a specific VRF EVPN
V V V V
• Sub-interface used for eBGP peering and as next-hop Leaf
• Per VRF, Per Sub-interface eBGP session

VXLAN EVPN
VRF-LITE using L3 Routed Interfaces
Clear separation of Autonomous Systems
External Network
Simple, Straight forward, and Commonly used
No need for redistribution

Edge Router VRF VRF VRF
Easy and Flexible BGP route-filtering mechanisms “A” “B” “C”
BGP natural loop avoidance

A B C
Structured handoff between the VXLAN BGP EVPN fabric and the Border Leaf
external routing domain (Backbone, WAN, Campus, etc.) V
Not ideal for High scale VRF handoff deployment R R
Spine
Peering Type = Sub-interfaces on physical routed (or L3 Port-
channel) interfaces V
V V V
Leaf
• Sub-interface with dot1q tag to mark the traffic to a specific VRF
• Sub-interface used for eBGP peering and as next-hop VXLAN EVPN
• Per VRF, Per Sub-interface eBGP session
VRF-LITE using L3 VLAN Interfaces (SVI)
Clear separation of Autonomous Systems
External Network
No need for redistribution
Easy and Flexible BGP route-filtering mechanisms

Edge Router VRF VRF VRF
BGP natural loop avoidance “A” “B” “C”
Not ideal for High scale VRF handoff deployment

A B C
Uncommon design approach Border Leaf
V
Peering convergence dependent on L2 (SVI) interfaces as well
R R
Peering Type = L2 Trunk Interfaces with SVIs Spine
• Physical interface configured as 802.1Q trunk port

V V V V
Leaf
• SVI peers per VRF basis
• SVI serves as next hop routing VXLAN EVPN
Seamless Protocol Gateway Model (Border-PE)
Combines two different encapsulations and
Address Family, using a “single-box (Border- Seamless Stitching between
PE)” instead of a “two-box (CE-PE)” model
VXLAN, MPLS, and SR
VXLAN VTEP Border nodes also becomes a
External Network
MPLS L3VPN Provider Edge (PE), resulting in
a role called Border-PE
Remote PE BGP 65099
Best suited for high scale VRF deployment
VPNv4/VPNv6
Saves CAPEX and OPEX MPLS-LDP or SR-MPLS
Seamless stitching between L2VPN EVPN
and VPNv4/v6 Address Family Reoriginate EVPN Prefix in L3VPN V V
Border-PE
Reoriginate L3VPN Prefix in EVPN BGP 65001
BGP route-filtering mechanisms available
Specific Hardware support R R

Spine
• MPLS LDP: Nexus 3600-R, Nexus 9500-R EVPN
V V V V
• SR MPLS: Nexus 9300 FX2/FX3/GX/GX2, Leaf
Nexus 9500-R
VXLAN EVPN
IGP (OSPF)
IGP (OSPF) External IP
Complex and Operationally difficult Handoff
IGP not recommend for Inter-Domain routing
VPNv4/VPNv6 External Network
Complex and cumbersome Route Redistribution (IGP <-> BGP MPLS-LDP or SR-MPLS
EVPN)
Edge Router BGP 65099
IGP (OSPF) External routes are preferred over EVPN Internal
routes due to Protocol Administrative Distance
IPv4/IPv6 (OSPF)
No natural protection from Routing Loops
V V
Limited IGP route filtering capabilities Border Leaf BGP 65001
Limited usage and not commonly deployed method

R R
Spine
EVPN
V V V V
NOTE: Form more information visit BRKDCN-2267 Leaf
VXLAN EVPN
Provision External IP
Handoff with NDFC
VXLAN EVPN External L3 connectivity
Prerequisites and Guidelines
Data Center VXLAN EVPN
Extend Layer-3 services to Nexus 9000, Nexus 7000, Catalyst
9000, ASR 1000, ASR 9000, ASR 8000, NCS S500 Spine
Supported role for Easy Fabric: Border, Border Spine, Border

Gateway, Border Gateway Spine, Border Super Spine, and Border
Gateway Super Spine
Leaf
NDFC Auto deploys VRF-LITE configurations when Edge device is a
Border Leaf
Nexus
*Meta switch in External Network supported

*A common case when user does not import destination Edge router in NDFC. In this case just create
an empty External fabric.
Edge Router
Layer-3 Ethernet and Port-channel supported
External Network
Inter-Fabric Connections (IFC) considerations
• Physical Ethernet Interface or Layer 3 Port-channel supported
• VRF-Lite IFC Deployment Type options:

• Manual (define IFC links manually) [Default]
• Use this option to create Links (UI: Fabric overview > Links > Create) when Edge device is Non-Nexus or
Unmanaged
• Back2Back&ToExternal (Multi-Site + VRF-LITE)
• IFC links automatically defined wherein Border/Border Gateways are directly connected to each other
• IFC links automatically defined on each Border physical interface connected to a device with Edge Router role in
the External Network fabric
• VRF can be extended to Multi-Site + VRF-LITE only if the device role is Border Gateway (or any variation of BGW
role). For Border-only roles, simply extend VRF as VRF-LITE only
Route Advertisement considerations
A per VRF per peer eBGP peering session between Border-Edge Router is established
for each VRF-Lite IFC. Three options are natively supported for this peering:
• Advertise Host (disabled by default in NDFC)
• If host routes (/32 or /128) need to be enabled and advertised from Border leaf to Edge router
• Route-map does outbound filtering
• Advertise Default-Route (enabled by default in NDFC)

• Controls whether a network statement 0/0 will enabled under the VRF
• Config Static 0/0 Route (enabled by default in NDFC)

• Controls whether a static 0/0 route to the Edge router should be configured under the VRF on the border leaf
• Advertise Host (disabled) • Advertise Host (enabled)

ip prefix-list default-route seq 5 permit 0.0.0.0/0 le 1 ip prefix-list default-route seq 5 permit 0.0.0.0/0 le 1
ip prefix-list host-route seq 5 permit 0.0.0.0/0 eq 32
route-map extcon-rmap-filter deny 10 route-map extcon-rmap-filter-allow-host deny 10
match ip address prefix-list default-route match ip address prefix-list default-route
route-map extcon-rmap-filter deny 20
match ip address prefix-list host-route
route-map extcon-rmap-filter permit 1000 route-map extcon-rmap-filter-allow-host permit 1000

vrf corp vrf corp
address family ipv4 unicast address family ipv4 unicast
send-community both send-community both
route-map extcon-rmap-filter out route-map extcon-rmap-filter-allow-host out
• Advertise Default-Route (disabled)
router bgp 65001

vrf corp
address family ipv4 unicast
advertise l2vpn evpn
maximum-paths ibgp 2
• Advertise Default-Route (enabled)

A 0/0 route is advertised inside the fabric over EVPN Route type 5 to the leafs,
thereby providing a default-route out of the leafs toward the border devices
router bgp 65001
vrf corp
address family ipv4 unicast
advertise l2vpn evpn
maximum-paths ibgp 2
network 0.0.0.0/0
• Config Static 0/0 Route (disabled) • Config Static 0/0 Route (enabled)
vrf context corp vrf context corp
description CORP description CORP
vni 9999 vni 9999
rd auto rd auto
route-target both auto evpn route-target both auto evpn
ip route 0.0.0.0/0 10.33.0.1

vrf corp vrf corp
send-community both send-community both
Enabling VRF-LITE Automated
Border Leaf to Nexus Edge Router
Step 2
IFC prototypes
Define IFC Type: “Back2Back&ToExternal”
Check: Auto-Deploy both
(Optional) Auto Allocation of Unique IP on VRF Extension over VRF LITE IFC
Recalculate config and deploy
Step 1 Step 3
Create and Import Deploy Symmetric VRF-Lite Extension
Define VXLAN EVPN and External Fabric. Set Define Individual VRF extension on the Border leaf
respective roles (e.g., Border, BGW, Edge)
NDFC will add Sub-int and eBGP peering on Border leaf and neighbor
Uncheck Fabric monitor mode in External Edge router
Fabric if the Edge router is in managed mode
Deploy on Data Center VXLAN EVPN Fabric and External Network Fabric
VRF-LITE: Border to Nexus Edge
Topology and IFC considerations
External Network
ASN 65100
VRF
“CORP”
• IFC Automated
• Advertise Host (disabled) Edge Router

”Nexus 9000”
• Advertise Default-Route (enabled) E1/3.2 – 10.33.0.1/30
• Config Static 0/0 Route (enabled) “Nexus 9000”

E1/3.2 – 10.33.0.2/30
Border Leaf
VRF
“CORP”
ASN 65001
VXLAN EVPN
Managing Edge Devices
part of “External Connectivity Network” Fabric
• Create Fabric, Discover Switches, Set Role, and Recalculate & Deploy
Defining IFC deployment type
• Review Fabric Settings for VRF-Lite IFC deployment type:
LAN > Fabrics > Select (your DC VXLAN Fabric) > Actions > Edit Fabric > Resources Tab
Per VRF Per Dotq1
association
Select
Deployment Type
Deploy VRF-LITE if
Edge device is
Nexus and
managed by NDFC
eBGP peering
subnet details
Defining IFC Link on physical interface
• IFC link has been defined
• Policy should be ext_fabric_setup
o IPs auto selected from VRF-Lite Subnet IP Range
Defining VRF extensions on Border
• Verify DC VXLAN Fabric VRFs were created and customize if needed

• LAN > Fabrics > Double click (your DC VXLAN Fabric) > VRFs
Edit VRF if you want to modify

route advertisement or other
VRF specifics
Define according to your

use case
• Attach VRF to Border and extend through VRF-Lite
Extend Options for Border
Extend Options for Border Gateway
Physical Interface
MTU
Provide VRF name for

External Edge. By default,
NDFC uses the same name
Extension Destination Edge
as Border VRF
Type router details
Provide Route-MAP
if different from
NDFC default
Preview and Deploy VRF extensions on Border
L3 Tenant
CORP Advertisement of Default route
towards site internal VTEP
L3VNI VRF
Configs
eBGP External
Edge neighbor
Default route static
towards External Edge
Border Physical
Interface Config
Preview and Deploy VRF extensions on External Edge
• Once configurations are deployed on Border Leaf (DC VXLAN EVPN), navigate to
External Network Fabric and perform Recalculate and Deploy
VRF Configs
eBGP Border
Leaf neighbor
Edge Physical
Interface Config
Enabling VRF-Lite: Manual
Border Leaf to Non-Nexus Edge Router
Step 2 Step 4
IFC prototypes
External Network Fabric VRF-Lite Extension
Define IFC Type: “Manual” Define sub-interfaces, invoke BGP policies
Define IFC link on physical interface NDFC will generate Sub-int and eBGP peering on Edge router
Recalculate config and deploy Deploy on External Network Fabric
Step 1 Step 3
Create and Import DC VXLAN EVPN Fabric VRF-Lite Extension
Define VXLAN EVPN and External Fabric. Set Define Individual VRF extension on the Border leaf
respective roles (e.g. Border, BGW, Edge)
NDFC will generate Sub-int and eBGP peering on Border leaf
Uncheck Fabric monitor mode in External
Deploy on Data Center VXLAN EVPN Fabric
Fabric if the Edge router is in managed mode
For Non-Nexus ensure SNMP configs for
discovery
VRF-LITE: Border to Non-Nexus Edge
Topology and IFC considerations
External Network
ASN 65099
VRF
“CORP”
• IFC Manual
• Advertise Host (disabled) Edge Router

”ASR 9000”
• Advertise Default-Route (enabled) Gig0/0/0/0.2 – 10.10.10.1/31
• Config Static 0/0 Route (enabled) “Nexus 9000”

E1/1.2 – 10.10.10.0/31
Border Leaf
VRF
“CORP”
ASN 10
VXLAN EVPN
Defining IFC deployment type
• Review Fabric Settings for VRF-Lite IFC deployment type:
LAN > Fabrics > Select (your DC VXLAN Fabric) > Actions > Edit Fabric > Resources Tab
Per VRF Per Dotq1

association
Select
Deployment Type
eBGP peering
subnet details
• Verify link from DC VXLAN Fabric to External Fabric was discovered
LAN > Fabrics > Double Click (your DC VXLAN Fabric) > Links
Use filters when having multiple fabrics
If no policy exists, or
neighbor not found. A link
can be created manually
• If Link discovered, values are pre-filled

• When not, make sure:
• Link Type is Inter-Fabric
• Link Sub-Type is VRF_Lite
• Link Template: ext_fabric_setup
External Fabric
“WAN-EXT”
ASN 65099
Edge Router
Gig0/0/0/0.2 – 10.10.10.1/31
E1/1.2 – 10.10.10.0/31
Border Leaf
ASN 10
DC Fabric
“VXLAN”
• Verify DC VXLAN Fabric VRFs were created and customize if needed

• LAN > Fabrics > Double click (your DC VXLAN Fabric) > VRFs
Edit VRF if you want to modify

route advertisement or other
VRF specifics
Define according to your

use case
Extend Options for Border
Extend Options for Border Gateway
Preview and Deploy VRF extensions on Border
Default route static

towards External Edge
Advertisement Default
route towards site internal
eBGP External
Edge neighbor
Border Physical
Interface Config
Defining the policies on the Edge Router
• After completing the configurations on the VXLAN Fabric (Border Leaf), Navigate to
External Fabric (Edge Router) and apply the following policies
• ios_xr_base_bgp
Policy = Ios_xr_base_bgp
Defining the policies on the Edge Router
• After completing the configurations on the VXLAN Fabric (Border
Leaf), Navigate to External Fabric (Edge Router) and apply the
following policies
• ios_xr_base_bgp
• ios_xr_Ext_VRF_Lite_Jython
Policy =
External Fabric Ios_xr_Ext_VRF_Lite_Jython
“WAN-EXT”
Edge Router
Gig0/0/0/0.2 – 10.10.10.1/31
E1/1.2 – 10.10.10.0/31
Border Leaf
ASN 10
DC Fabric
“VXLAN”
Deploy configs on the Edge Router
IOS-XR eBGP Policy

for allowing routes
IOS-XR VRF Definition
eBGP Border
Leaf neighbor
IOS-XR Physical
Interface Config
Verification and
Validation with
NDFC
Verification through NDFC
Keeping you away from CLI
Step 2
Deployment History
Configuration Execution Status:
Verify Deployment History Status Success for
Underlay, Overlay, Interfaces, and more
Step 1 Step 3
Verify Network and VRF Show commands
attachments Service / features status
Job Execution Status: (CLI through NDFC)
Network Status Deployed
VRF Status Deployed
Attachment deployment status
Job execution perspective
Success or Failure deployment details
Multi-Stage Preview and Deployment
Deployment History Tool
Commands execution perspective
CLI response messages for

easier troubleshooting
Show Commands Tool
Switch Config perspective
NDFC pre-built
commands, user
commands, or custom-
built show templates
Key NDFC Template Terminology
• Template: CLI code-chunks of CLI features, with variables and default values, that go
from VXLAN EVPN fabrics up to the Overlay networks, VRFs, and much more, like PBR,
vPC pairing, BGP, MPLS, etc.
• NOTE: Do Not edit/modify shipping templates. Instead, duplicate the existing template to
change the contents and provide a unique name.
Template Types
• Broadly used to configure devices features (hence, the Policy Template reference
Policy: everywhere in NDFC)
• Overlay Mode, config-profile mode for VRFs and Networks. where the overlay
Profile: dependencies are managed in the same place (BGP, vlan, vni, intfs, vrf, etc)
• Overlay Mode, can be used instead of Profile. Config is pushed in a “traditional”

CLI: CLI format
• The “heavy-lifter” special python template, manages the fabric global parameters
Fabric: and the Policy and Profile/CLI Template Intents for each fabric device.
• Allows pushing a bulk of show commands to devices in order to verify services

Show: status.
• Used to support the Programable Reports feature, a python SDK bundled with
Report: NDFC, provided to simplify report generaton.
• To use commands available in EXEC mode, includes show commands, clear commands,
Exec and other commands that perform actions that you do not save in the device configuration.
Policy Template Instance (PTI)
• When a Policy Template is attached to a device, a unique Policy ID is generated, contains details,
variable values and the expected generated configuration of that template that will be applied to
the device.
Policy
Manager
Priority order, lowest

number gets executed first
Policy
Templates
Day-2 with
NDFC
Deployment History Tool
Who deployed what, when, where?
CLI response messages for

easier troubleshooting
Streamlined Image and Patch Mgmt. for NX-OS
Simplify, Speed up, and Mitigate Risks of Errors
Upgrading manually might take a long time and

Why prone to error
Image Management Automates the whole

How Upgrade process.
Automate Software Upgrade for One or a Group

What of Cisco Nexus switches
When Scheduled & On-demand
Image Management Workflow with NDFC
1 2
CCO
End-user Image Policy
Intent 3
Workstation
NDFC Mgmt OOB Mgmt
6
4
NDFC
Discover the switches into Nexus Dashboard Fabric Controller.
1 2 3 4 5 6
Validate non-
Upload Create the Attach the Stage the Upgrade the
disruptive
Images Image Policy Image Policy Images switches
upgrade
Return Material Authorization (RMA)
End-user 4
Workstation
3 NDFC Mgmt OOB Mgmt
1 2
NDFC
Restore Config
1 From the Fabric Overview, select the concerned switch and Change its Mode to Maintenance
2 Replace the switch with new one
3 Provision RMA and Set the Admin Password (for POAP). Manual RMA is also supported
4 Select the device to replace and click Provision RMA
5 Move the device back to Normal Mode (~5 mins for full configuration)
Granular Role-Based Access Control (RBAC)
Orchestrated from ND
Network Access Administrator
Administrator (Full (rights for interfaces
admin rights) management only)
Enhanced RBAC &
Security Domains
Network
Device Upgrade Login Authentication (Local &
Administrator (rights
Operator (read
only/preview)
for image Remote)
management only)
Associate a site to a Security Domain
Network Associate a user to a security

Stager (stage
config/policies on Domain
NDFC only)
Supports same user to have different

roles for different fabrics
Programmable Reports
NDFC Dashboard
Overall
Fabric Switches
health Overall configuration
existing status
Alarms and
Events
Switches
inventory by
Switches number of
operational roles
status
Programmable
reports for Image Switch inventory
management and based on different
Performance hardware and
monitoring software versions
NDFC Dashboard
Allow network administrators to focus on daily operation around the health, sync
configuration, and performance of data center switching.
Multiple additional Tabs based on the Features enabled (EPL, VMM, K8s)
Visualize multiple fabrics with topology views
Underlay
and
Overlay
links
VXLAN Switch
Overlays specific
Oper vs information
Config view
Color coded
Compliance
status
NDFC Topology
Operational Color Coding
Operation Configuration
The element is in good health and functioning as intended.
The element is in warning state and requires attention to prevent any further problems
The element has minor issues
The element has major issues and requires attention to prevent any further problems
The element is in critical state and requires immediate attention
The element has not been discovered
NDFC Topology
Configuration Color Coding
Operation Configuration
The element is In-Sync with the intended configuration.
The element has pending deployments.
Indicates that active deployments are in-progress.
The element is Out-of-Sync with the intended configuration.
No support for Configuration Sync/not discovered
NDFC Topology
Switch Logical Links
• Troubleshoot Physical v Logical peering’s

• Routing Protocol state details
• Physical link details such as IP prefix,
Interface speed, Interface mode, and
more…
NDFC Topology
ngOAM Source & Destination Physical Path of a particular flow Host to Host path
Topology Menu ☞ Actions ☞ VXLAN
OAM Switch to Switch path
Live path Monitoring & Troubleshooting

visualization
Live path
visualization
Monitoring Alarm
Viewing Alarms and Events
Alarms generated for

various categories
History of Cleared Alarm
Manage Alarm Policies
Monitoring Alarm
Create & Activate new Alarm Policies

Device Health Policy
Monitoring Alarm

Interface Health Policy
Monitoring Alarm

Interface Health Policy
Syslog Alarm Policy
Monitoring Alarm
Events generated for the

switches
Display Information
Acknowledge Delete Events
Monitoring Alarm
Accounting Information
Delete Events
NDFC Compute Visibility
Virtual Machines Manager Integration (Read Only) and Visibility
Virtual Management Menu ☞ Virtual Infrastructure Manager Tab
Add the Instance of choice: vCenter, K8s cluster or OpenStack cluster

with its reachability information as well as its credential
Expand the view to compute hosts, virtual switches and Virtual Machines.
Select any VM and get Networks details up to the Leaf node
From the topology view, enter into the VMM instance to view the Hosts,
Virtual Switches, and Virtual Machines
Compute Visibility
Display
connected
Physical Hosts
Display
DVS/vSwitch
Display
VMs
Benefit
Single point of management providing in depth visibility and
VMM, Kubernetes, and OpenStack VM level visibility
information
VMM Visibility vCenter Instance
Visualise the Virtual Infrastructure at different layer
Hosts DVS VM Display connectivity details
Leaf nodes
Physical Host
DVS
Virtual Machines
K8s Visibility K8s Instance
Visualise the K8s Infrastructure at different layer
Display connectivity details

Nodes PODs
Leaf nodes
Physical Host
DVS
K8s Node
Namespace
POD
Endpoint Locator and Visualizer
NDFC 12 RR RR
Mgmt Interface
ND cluster
Data Interface IB Mgmt
EndPoint Locator relies on Fabric Inband Mgmt to collect EP information, using the ND Data Interface
An endpoint is typically anything with one IP address (IPv4 and\or IPv6) and/or MAC address
NDFC pre-provisions the switches that host the MP-BGP Route Reflector function to peer with them
NDFC contacts the selected RRs and add the appropriate BGP neighbor statements on the RR (Spine)
NDFC 12 RR RR
Mgmt Interface
ND cluster
Data Interface IB Mgmt
The Endpoint Locator (EPL) feature allows Real-time Tracking of Endpoints within a Data Center
Tracing the Network Life History of Endpoints
Analyze trends associated with endpoint such as additions, removals, moves, and so on
Overlay
count
EPs tracking and

availability
EPs
Active EPs in a connectivity
fabric
EPs per
switches
EPs detailed
information
Benefit
Almost real-time data on active Single pane of glass for all EP Endpoint life and history Active VRFs and Networks
endpoints visualization
NDFC Service Backup & Restore
NDFC Service Backup VXLAN Fabric
LAN Fabric
Multi-Site Domain (MSD)
NDFC Service Backup & Restore
Config Only Config Only
Full Backup Full Backup

On-Demand Scheduled
Backup Backup
Remote Server (SCP) New NDFC App

Local Workstation Config only or Full
Backup
Restore
Location
• Configuration Backup: Config Intent, Discovery, Credential, and Policies.
• Full Backup: Configuration Backup

charts.
+ historical data like reports, metric
NDFC with Day 2 operations
Seamless integration with Day 2 operations app NDI for in depth
telemetry analytics
Management Real-time insights
Benefit
Advanced monitoring and troubleshooting capabilities
Cisco Nexus Dashboard Insights
360-degree visibility Analytics Integrate with NDFC
Security
On-prem
resources
Cloud
Other services
Networking
apps
NDI NDFC
ITSM
Network, Digitization, Unprecedented

applications, integrations telemetry, correlation visibility into data
center environments
Nexus Dashboard Insights
Identify, Locate, Upgrade impact
Use Cases Root cause, Remediate Advisories
Error detection, latency,

Mitigate
Packet drops
Prevent outages
Control plane issues
Automated alerts Hardening checks

Software Hardware
Visibility recommendations
Capacity alerts PSIRT notices

Environmental alerts EoS/EoL notices
Correlation TAC assist

Statistical models Topology checker
Easily license your NDFC
Software included with switch subscription licenses
Tier-based
NXOS Essentials, Advantage, Premier
DCN Essentials, Advantage, Premier
Install base (NX-OS) New purchase (NX-OS)
Increased ease of use Less restrictions with Reduced overall licensing

and efficient consumption flexible reporting friction for users
NDFC 12.0 is transitioning to a Smart Licensing only release

Appendix
White Paper and Document References
• VXLAN BGP EVPN

• https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-
switches/white-paper-listing.html
• NDFC
• https://www.cisco.com/c/en/us/products/cloud-systems-
management/prime-data-center-network-manager/white-paper-
listing.html
Conclusion
Key points to remember
AZ-1 AZ-4
AZ-2 AZ-5
AZ-3 AZ-6
• VXLAN EVPN Multi-Site maintains clear change and fault domain

separation to deploy large-scale and highly available DC
architectures.
• NDFC provides simplified mechanism to extend and provide

end-to-end network and policy consistency across regions, all
with a single point of orchestration.
• NDFC provides a single plane of glass solution to automate and

manage Nexus and Non-Nexus devices
Thank you

BRKDCN 2988

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 2988

Uploaded by

Copyright:

Available Formats

Design, Automate, and

Manage VXLAN BGP EVPN

Parth Patel, Technical Leader, Technical Marketing Engineer:

Agenda • Automate and Manage

EP1 VLAN EP2 EP3 VLAN EP4

EP1 VLAN EP2 EP3 VLAN EP4

EP1 VLAN EP2 EP3 VLAN L2

EP1 VLAN EP2 EP3 VLAN L2

ip access-group web2 out

VLAN EP2 VLAN EP4L2 L3

Layer-2 STP Forwarding

STP Root STP 2nd Root Layer-2 STP Blocked

Layer-2 STP Forwarding

STP Root STP Root

VPC and Spanning-Tree

Anycast HSRP Anycast HSRP

Hierarchical Topology Hair-Pining Flood & Learn

• BGP EVPN must be used in the overlay to

• VXLAN is used to transport endpoint traffic Underlay

Hierarchical Topology No More Hair-Pining Control-Plane Learned

Spine Spine Spine Spine

• Single Replication Domain for BUM

• Single Overlay Domain – End-to-End

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

“The Data Center of Yesterday”

• Multiple Underlay Domains - Isolated

• Multiple Overlay Domain – Interconnected and

• Multiple Overlay Control-Plane Domains –

• Multiple VNI Admin Domain – Downstream VNI

“The Data Center of Today”

Network Fault Domain Network Fault Domain

Application Change Domain

• Packet Re-Origination (L2 and L3)

• Inter-Site DCI (East-West)

• L3 Extension (North-South) IP Handoff

• Connect L4-L7 services and EPs

• Integration with Legacy Networks (Co-existence and/or Migration) VXLAN EVPN

• VXLAN L3-extension to the Public Cloud

Anycast Border Gateway VPC Border Gateway

• Up to 6 BGW • 2 BGW with Physical Peer-Link

Flexible Anycast or VPC BGW models

Capacity planning only for DCI traffic flows Border Gateway

Border Gateway hosts: R R

- East-West (DCI Packet Re-Origination L2/L3)

Extra functional dependency (BGW + Spine)

- East-West (DCI Packet Re-Origination L2/L3)

V V = VTEP VXLAN EVPN Site

Architecture beyond a single server room. Simpler capacity Border Gateway

Clean role separation and uniform reachability from the entire R R

- East-West (DCI Packet Re-Origination L2/L3)

Capacity planning needs to accommodate all flows Border Gateway Super-Spine

Extra functional dependency (BGW + Super-Spine). Not V V

Border Gateway Super-Spine hosts:

VXLAN EVPN Site

• Backbone/Cloud can be any routed service. (flat L3,

• Multi-Site Underlay: eBGP IPv4 Unicast

• Site-Internal Fabric BUM: Ingress-Replication or

• Minimum topology is the square. An enhanced

• Multi-Site Underlay: eBGP IPv4 Unicast

• Site-Internal Fabric BUM: Ingress-Replication or

Site-ID: 65001 Site-ID: 65002

• DF election Formula uses “mod” operation:

• I = V mod N {I = ordinal value, V = VLAN #, N = # BGWs in a site}