#CiscoLiveAPJC

Introduction to VXLAN
The Future Path of Your Data Center

Richard Licon
Principal Technical Marketing Engineer
BRKDCN-1621

#CiscoLiveAPJC
#CiscoLiveAPJC
Cisco Webex App

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App

2 Click “Join the Discussion”

3 Install the Webex App or go directly to the Webex space Enter your personal notes here

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

by the speaker until December 22, 2023. https://ciscolive.ciscoevents.com/ciscolivebot/#BRKDCN-1621

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Session Abstract
• The need for flexibility in the data center opens the door to network
overlays. Several standards have been proposed and implemented to
enable virtualized networks to be layered over a physical network. The data
center network needs to be flexible enough to support workloads that can
move from one host to another, and for new services to be deployed
rapidly. VXLAN is one such technology that provides massively scalable
virtual network overlays on top of existing IP infrastructures. Let's learn
more about VXLAN and delve deeper into it.
• This session aims at covering what is VXLAN, how it really works, and what
problems does it solve. This session will have configuration examples of
how we enable a VXLAN Fabric. No prior knowledge of VXLAN is required.
A basic understanding of Unicast and Multicast routing protocol is needed
along with an understanding of MP BGP

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 • Evolution of the Data Center
• The Need for network overlays
• What is VXLAN?
• The Need for a Control Plane

Agenda • Configuring EVPN VXLAN

• Packet Walks
• Conclusion

BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Evolution of the
Application Architecture
Evolution of the Application Architectures
Monolithic Web Applications Microservice Architecture

User Interface

Business Layer Application

Data Interface

Microservice Microservice Microservice Microservice

Database

Bare Metal
Virtualized Container Container Container Container
Compute

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Evolution of the
Data Center Network
 Evolution of the Data Center Network
Classic Spanning-Tree vPC and Spanning-Tree

Core Core Core Core

STP Root STP 2nd Root STP Root STP 2nd Root
FHRP Active
Agg Agg FHRP Standby FHRP Active
Agg Agg FHRP Standby

vPC

Access Access Access Access Access Access

vPC

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Evolution of the
Data Center
Network Architecture:
Clos Networks
Evolution of the Data Center Network
3-Stage Clos Fabric

Input Leaf Leaf Leaf Leaf Ingress

Stage

Middle Spine Spine Spine

Middle
Stage

Output Leaf Leaf Leaf Leaf Egress

Stage
Invented by Edson Erwin in 1938
March 1953: A Study of Non-Blocking Switching Networks. (Clos, Charles)
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Evolution of the Data Center Network
3-Stage Fabric

Spine Spine Spine Spine

Leaf Leaf Leaf Leaf

POD1 POD2

Virtualized Physical Virtualized Physical

Host Host

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Evolution of the Data Center Network
5-Stage Fabric
Super Super
Spine Spine

Spine Spine Spine Spine

Leaf Leaf Leaf Leaf

POD1 POD2

Virtualized Physical Virtualized Physical

Host Host

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Evolution of the Data Center Network
FabricPath (MAC-in-MAC) VXLAN Flood & Learn (MAC-in-IP)

Core Core Core Core

Anycast HSRP Anycast HSRP FHRP Active FHRP Active

Spine Spine “Spine” “Spine”

vPC

Leaf Leaf Leaf Leaf Leaf Leaf

vPC vPC

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Data Center Network Challenges
Legacy Architectures
Suboptimal Forwarding:
Path defined by
Convergence Time:
Root Switch or FHRP Active
STP high convergence time
TCN MAC Flush

OpEx/CapEx:
Unused Links: Expensive to upgrade
Scale up with large Chassis
STP Redundant links in
blocked state

Limited Scale:
Rigid Network Services: No Control Plane
L4-L7 Services placed at Limited Workload Mobility
Distribution Layer

“Spine” = Not Really a Spine #CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
The Need for
Network Overlays
Evolution of the Data Center Network
The need for network overlays

Spine Spine Spine Spine

North-South
Leaf Leaf Leaf Leaf Leaf

Container
Storage
Container
Bare Metal Storage
Container
Storage

East-West
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Overlay Taxonomy

Underlay
Overlay Services Tunnel Encapsulation
Network Transport

Control-Plane Data-Plane

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Overlay Services

Underlay
Overlay Services Tunnel Encapsulation
Network Transport

Control-Plane Data-Plane

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Overlay Services

Overlay Services

Layer-2 Layer-3 Layer-2 & Layer-3

Bridging Integrated Routing &

Routing Bridging (IRB)
Pseudo-Wire

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Tunnel Encapsulation

Underlay
Overlay Services Tunnel Encapsulation
Network Transport

Control-Plane Data-Plane

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Tunnel Encapsulation

Tunnel Encapsulation

Layer-2 Layer-3

MPLS L2VPN MPLS L3VPN

Q-in-Q GRE
NVO3 (VXLAN) LISP

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Underlay Network Transport

Underlay
Overlay Services Tunnel Encapsulation
Network Transport

Control-Plane Data-Plane

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Underlay Network Transport

Underlay
Network Transport

Layer-2 Layer-3

IS-IS*
IS-IS
OSPF
STP
BGP
*IS-IS operates at L2 but transports L2 & L3
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Control-Plane

Underlay
Overlay Services Tunnel Encapsulation
Network Transport

Control-Plane Data-Plane

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Control-Plane

Control-Plane

Learning Route Distribution Peer Discovery

Local Flood & Learn Flood & Learn

Remote BGP BGP

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Data-Plane

Underlay
Overlay Services Tunnel Encapsulation
Network Transport

Control-Plane Data-Plane

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Data-Plane

Data-Plane

Encapsulation Unicast Forwarding BUM

(Broadcast, Unknown Unicast, Multicast)

Imposition (Encapsulate) Layer-2 Unicast-based*

Disposition (Decapsulate) Layer-3 Multicast-based**
*Ingress/Head-End Replication
**PIM #CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
VXLAN
Virtual Extensible LAN
What is VXLAN?

Standards-
Transport UDP Flexible
based
Independent Encapsulation Namespace
encapsulation

Layer-3
Multi-path
RFC 7348 Transport Segmentation
capable
(Underlay)

Per Flow
MAC-in-IP
Entropy

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Underlay Taxonomy
• Edge Devices host the VTEP
• Responsible for the encapsulation and decapsulation of the VXLAN Header

V VTEP: VXLAN Tunnel Endpoint

NVE: Network Virtualization Edge

Edge Devices
Edge Device
V (NVEs)
(NVE)

V
Virtualized
Hosts
Hosts
(Endpoint)
(Endpoint) (Physical/Virtual)
(Physical/Virtual) Virtualized
V

Underlay
Control-Plane Underlay Transport Network
Physical
Host

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 Overlay Taxonomy
Overlay Control-Plane
V

Service = Virtual Network V

Identifier = VN Identifier (VNI/VNID) Tunnel Encapsulation
V VTEP: VXLAN Tunnel Endpoint V
NVE: Network Virtualization Edge

Edge Devices
Edge Device
V (NVEs)
(NVE)

V
Virtualized
Hosts
Hosts
(Endpoint)
(Endpoint) (Physical/Virtual)
(Physical/Virtual) Virtualized
V

Underlay
Control-Plane Underlay Transport Network
Physical
Host

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
VXLAN (or Virtual) Tunnel Endpoint
VTEP
VLAN
100

Virtualized
V
VTEP VNI

VXLAN
Virtualized
Ethernet

• VXLAN Tunnel Endpoint - Network Virtualization Edge

• Each VTEP is uniquely identified by an IP Address
• VTEP Discovers or learns remote VTEPs, and end hosts attached to them
• VTEP bridges when forwarding packets within the same VNI and Routes
for Inter-VNI traffic

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
VXLAN Encapsulation / Packet Format
VLAN
100

Virtualized
V
VTEP VNI

VXLAN
Virtualized
Ethernet

Outer Optional Outer Optional Original

Outer Outer UDP Inner Inner
MAC Outer VXLAN Inner Ethernet CRC
IP SA IP DA (Port MAC SA MAC DA
SA/DA 802.1q 4789) 802.1q Payload

VXLAN Encapsulation Original Ethernet Frame

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
VXLAN Packet Format
• VXLAN uses MAC in UDP encapsulation

IP/UDP/VXLAN
Outer Optional Outer
Outer Outer UDP
MAC Outer VXLAN
IP SA IP DA (Port
SA/DA 802.1q 4789)

14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes

VXLAN Encapsulation (50/54 Bytes)

• Adds UDP and VXLAN Header before original Ethernet Frame

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
VXLAN Header Details
Flags 1 Byte
RRRRIRRR (8 bits)

IP/UDP/VXLAN
1 Byte
Reserved
(8 bits)
Outer Optional
Outer Outer Outer
MAC Outer VXLAN
IP SA IP DA UDP
SA/DA 802.1q
3 Bytes
VXLAN Network Identifier (VNI)
14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes
(24 bits)

VXLAN Encapsulation (50/54 Bytes) 3 Bytes

Reserved
(24 bits)

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 VXLAN Header Details
Flags 1 Byte
RRRRIRRR (8 bits)

IP/UDP/VXLAN
1 Byte
Reserved
(8 bits)
Outer Optional
Outer Outer Outer
MAC Outer VXLAN
IP SA IP DA UDP
SA/DA 802.1q
3 Bytes
VXLAN Network Identifier (VNI)
14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes
(24 bits)

VXLAN Encapsulation (50/54 Bytes) 3 Bytes

Reserved
(24 bits)

§ Flags Field: I-flag (set to 1) for valid VNI. Other flags remain as R (set to 0)

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 VXLAN Header Details
Flags 1 Byte
RRRRIRRR (8 bits)

IP/UDP/VXLAN
1 Byte
Reserved
(8 bits)
Outer Optional
Outer Outer Outer
MAC Outer VXLAN
IP SA IP DA UDP
SA/DA 802.1q
3 Bytes
VXLAN Network Identifier (VNI)
14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes
(24 bits)

VXLAN Encapsulation (50/54 Bytes) 3 Bytes

Reserved
(24 bits)

• Flags Field: I-flag (set to 1) for valid VNI. Other flags remain as R (set to 0)
• VNI Field: Allows VNI 1-16,777,215 (some implementation only 4096-16,777,215)

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
VXLAN Header Details
Flags 1 Byte
RRRRIRRR (8 bits)

IP/UDP/VXLAN
1 Byte
Reserved
(8 bits)
Outer Optional
Outer Outer Outer
MAC Outer VXLAN
IP SA IP DA UDP
SA/DA 802.1q
3 Bytes
VXLAN Network Identifier (VNI)
14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes
(24 bits)

VXLAN Encapsulation (50/54 Bytes) 3 Bytes

Reserved
(24 bits)

vlan 15
vn-segment 31234

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Transport Independence

IP/UDP MAC
Outer Optional Optional Original
Outer Outer Outer Inner Inner
MAC Outer VXLAN Inner Ethernet CRC
IP SA IP DA UDP MAC SA MAC DA
SA/DA 802.1q 802.1q Payload

14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes

VXLAN Encapsulation (50/54 Bytes) Original Ethernet Frame

Source Destination
VTEP IP VTEP IP

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Multipath Capable

IP/UDP MAC
Outer Optional Optional Original
Outer Outer Outer Inner Inner
MAC Outer VXLAN Inner Ethernet CRC
IP SA IP DA UDP MAC SA MAC DA
SA/DA 802.1q 802.1q Payload

14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes

VXLAN Encapsulation (50/54 Bytes) Original Ethernet Frame

Source Destination
VTEP IP VTEP IP

UDP Dest. Port: 4789

UDP Src. Port: Per-Flow
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Multipath Capable

IP/UDP MAC
Outer Optional Optional Original
Outer Outer Outer Inner Inner
MAC Outer VXLAN Inner Ethernet CRC
IP SA IP DA UDP MAC SA MAC DA
SA/DA 802.1q 802.1q Payload

14 Bytes 4 Bytes 20 Bytes 8 Bytes 8 Bytes

VXLAN Encapsulation (50/54 Bytes) Original Ethernet Frame

Source Destination
VTEP IP VTEP IP MAC SA/DA
IP SA/DA
Protocol
Port
UDP Dest. Port: 4789 UDP Source Port Algorithm is driven
UDP Src. Port: Per-Flow by the Hardware (Switch ASIC)
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Data Plane
Flood and Learn
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
(0100.5E01.0102) VTEP2

Leaf-1
IP: 10.200.200.1 VTEP1

Leaf-3
IP: 10.200.200.3 VTEP3

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
(0100.5E01.0102) VTEP2

Leaf-1
IP: 10.200.200.1 VTEP1

Leaf-3
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF

1 Host A Sends out an ARP Request

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
IP: 10.200.200.1 VTEP1
2 VTEP1 Receives ARP Request
Performs Layer 2 Lookup
based on VNI and DMAC Leaf-3
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
IP: 10.200.200.1 VTEP1
3 VTEP1
encapsulates
the packet Leaf-3
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 with VXLAN IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF
Header

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
4 VXLAN Encapsulated packet Multicast Group: 239.1.1.2
is sent over the IP Core over
the multicast tree
Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
IP: 10.200.200.1 VTEP1

Leaf-3
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
IP: 10.200.200.1 VTEP1 VTEP3 decapsulates VXLAN
5 packet Layer 2 Learning results
with the following entry:
Leaf-3
ARP Request for 192.168.1.102 MAC VNI VTEP
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF 0000.3000.1102 3001 Ethernet1/8

0000.3000.1101 3001 10.200.200.1

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
IP: 10.200.200.1 VTEP1 VTEP 3 performs
a Layer 2 Lookup
6
Based on VNI= 30001,
Leaf-3 DMAC = FFFF.FFFF.FFFF
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF VTEP 3 forwards the Broadcast
packet to all hosts part of
VNI 30001 (Host B)

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
IP: 10.200.200.1 VTEP1

Leaf-3
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF
ARP Response for 192.168.1.101
SMAC: 00:00:30:00:11:02
DMAC: 00:00:30:00:11:01

7 Host B sends out an

Host A Host B
MAC: 0000.3000.1101 ARP Response to VTEP3 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11

Leaf-1
MAC VNI VTEP
IP: 10.200.200.1 VTEP1
0000.3000.1102 3001 Ethernet1/8

0000.3000.1101 3001 10.200.200.1

Leaf-3
ARP Request for 192.168.1.102
SMAC:0000.3000.1101 IP: 10.200.200.3 VTEP3
DMAC: FFFF.FFFF.FFFF 8 VTEP3 receives the packet Performs
local learning and destination lookup
based on VNI = 30001
DMAC = 00:00:30:00:11:01
Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11
MAC VNI VTEP

Leaf-1 0000.3000.1102 3001 Ethernet1/8

IP: 10.200.200.1 VTEP1 0000.3000.1101 3001 10.200.200.1

9 VTEP3 encapsulates the packet

Leaf-3 with a VXLAN Header & forwards
IP: 10.200.200.3 VTEP3 it to VTEP1

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11
MAC VNI VTEP

Leaf-1 0000.3000.1102 3001 Ethernet1/8

IP: 10.200.200.1 VTEP1 0000.3000.1101 3001 10.200.200.1

VTEP1 Performs Layer 2 Learning

10 and destination lookup based on Leaf-3
VNI 30001, DMAC 0000.3000.1101 IP: 10.200.200.3 VTEP3

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
VNI: 30001
IP Subnet: 192.168.1.0
Multicast Group: 239.1.1.2

Spine Spine

Destination Group
239.1.1.2 Leaf-2
MAC VNI VTEP (0100.5E01.0102) VTEP2
0000.3000.1101 3001 Ethernet1/11
MAC VNI VTEP

Leaf-1 0000.3000.1102 3001 Ethernet1/8

IP: 10.200.200.1 VTEP1 0000.3000.1101 3001 10.200.200.1

11 ARP Response packet

Leaf-3
delivered to Host A IP: 10.200.200.3 VTEP3
ARP Response for 192.168.1.101
SMAC: 00:00:30:00:11:02
DMAC: 00:00:30:00:11:01

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 VXLAN Flood and Learn (F&L)
Host A to Host B Communication
• Host MAC Information is populated at both VTEPs (1&3)
• Host A & B know of each other’s MAC-IP Binding
• Subsequent trafficSpine
between Host A & Host B is unicast
Spine over
VXLAN between VTEP1 & VTEP3
Leaf-2
MAC VNI VTEP VTEP2
0000.3000.1101 3001 Ethernet1/11
MAC VNI VTEP

Leaf-1 0000.3000.1102 3001 Ethernet1/8

IP: 10.200.200.1 VTEP1 0000.3000.1101 3001 10.200.200.1

Leaf-3
IP: 10.200.200.3 VTEP3

Host A Host B
MAC: 0000.3000.1101 MAC: 0000.3000.1102
IP: 192.168.1.101 IP: 192.168.1.102
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Flood & Learn Mechanism
Multidestination Traffic is “Flooded” over the VXLAN Tunnel between VTEPs
to “Learn” about the Host MAC addresses located behind the VTEPs so
subsequent communication is delivered via Unicast

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 Flood & Learn Mechanism
Ingress Replication or Head-End Replication
• One unicast replica per remote VTEP
• Increases
Spine
traffic load throughout the network
Spine
• Example: 10MB of BUM traffic for 20 remote VTEPs = 200MB of BUM traffic

Ingress Replication
to each VTEP

Leaf-1
VTEP1

Leaf-2 Leaf-3 Leaf-4

ARP Request for 192.168.1.102
VTEP2 VTEP3 VTEP4 Leaf-20
SMAC:0000.3000.1101
DMAC: FFFF.FFFF.FFFF VTEP20

Host A Host B Host C Host D Host T

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Why VXLAN? - How did we get here?
Scale beyond 4k Segments Scale up to 16M Segments
(VLAN ID Limitation)

VM Mobility Seamless VM Mobility

Efficient use of bandwidth Leverages ECMP

VXLAN Delivers
(Blocked/Unused Links/ STP) (Path Optimization)

Workload Portability Any Workload Anywhere

(VLANs Limited by L3 boundaries) (Across L3 Boundaries)

Secure Multitenancy Traffic and Address Isolation

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Control Plane
EVPN
EVPN in the Data Center
IETF RFC/Draft for Control and Data Plane

Control-
EVPN MP-BGP RFC 7432
Plane

Multiprotocol Label Provider Backbone Bridges Network Virtualization Overlay

Data- Switching (MPLS) (PBB) (NVO)
Plane draft-ietf-l2vpn-evpn draft-ietf-l2vpn-pbb-evpn draft-ietf-bess-evpn-overlay

• EVPN over NVO Tunnels (i.e., VXLAN) for Data Center Fabric Encapsulations
• Provides Layer-2 and Layer-3 Overlays over IP Networks

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
 For your reference

BGP EVPN Route Types - RFC 7432

RFC/Draft Route Type Description

Ethernet Auto-Discovery EAD Route

MAC/IP Advertisement Route

RFC 7432
Inclusive Multicast Ethernet Tag Route

Ethernet Segment Route

draft-ietf-bess-evpn-
IP Prefix Route
prefix-advertisement

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 For your reference

BGP EVPN *Route Types - RFC 7432

RFC/Draft Route Type Description

BGP Based Multi-Homing

Ethernet Auto-Discovery EAD Route Mass Withdrawal/Aliasing

L2 VNI MAC or MAC-IP from

MAC/IP Advertisement Route L2 MAC Learning or ARP

RFC 7432
Dynamic Peer Discovery for
Inclusive Multicast Ethernet Tag Route EVPN Ingress Replication

BGP Based Multi-Homing

Ethernet Segment Route BUM DF Election/Split-Horizon

draft-ietf-bess-evpn- IETF Draft,

IP Prefix Route
prefix-advertisement Advertise IP Prefixes

*Type 6 - Selective Multicast Ethernet Tag Route

*Type 7/8 - IGMP Sync Routes
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
MP-BGP VPN Terminology
VPN Address Family
A Multi-Protocol BGP Extension to BGP Advertisement
Distribute VPN Routes Spine Spine = RD:10.1/16
VPN-IPv4 Address Spine Spine
BGP Next-Hop = NVE1
Route-Target = 10:10
Virtual Routing & Forwarding (VRF)
Overlay MAC or IP Routes are stored in
dedicated VRF tables (MAC or IP VRF)

VXLAN Tunnels
Route Distinguisher (RD) RD:10.1/16 V
NVE1Leaf
V Leaf
NVE3
8-byte Field, VRF Parameter
Value to make the MAC or IP Routes unique
(RD + VPN prefix) V RD:10.1/16
NVE2Leaf

Route Target (RT)

8-byte Field, VRF Parameter VRF Parameter: VRF Parameter:
Value for the Import/Export Rules of Name: VRF-Green
RD: 1:100
Name: VRF-Green
RD: 1:100
VPN Routes (MAC or IP) into the VRF Import RT: 10:10
Export RT: 10:10
Import RT: 10:10
Export RT: 10:10

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
EVPN in the Data Center
Control Plane Functions
RR RP
Host MAC/IP
RR RP
advertisements through
VTEP Peer Auto-Spine Spine MP-BGP
Spine Spine ARP
Discovery &
Suppression
Authentication via
MP-BGP
VRF-RED VRF-GREEN

V V V V
Leaf Leaf Leaf Leaf Ingress
Anycast IP Gateway Replication

Container
Virtualized
Container
Physical Virtualized
Container
Host

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Control-Plane Comparison
Flood&Learn (RFC7348) EVPN - Ethernet VPN (RFC8365)
• Ethernet over IP • A Better Ethernet over IP
• No Spanning-Tree (terminates at NVE) • No Spanning-Tree (terminates at NVE)
• Endpoint Learning is based on Flood and • Endpoint Learning is based on BGP
Learn (it’s in the name) exchange (EVPN uses BGP)
• Requires Extra Work for Routing • Provides Integrated Routing & Bridging
• FHRP for Default Gateway (IRB)
• Over-the-Top VRF-lite for Prefix Routing (or use • Distributed Anycast Gateway for Default Gateway
the Underlay?!) • Uses a Layer-3 VPN approach like MPLS L3VPN
• And there is much more in EVPN!

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Configuring the
EVPN VXLAN Fabric
 VTEP Device Roles

Spine: Interconnects
Leafs and Border Leafs Spine Spine Spine Spine

Border Leaf:
Leaf: External Connectivity
Hosts and Other
Devices Connect
V V V V V Border
Leaf Leaf Leaf Leaf
Leaf

Container Internet
Container
Bare Metal
Container

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 RR Route Reflector

VXLAN BGP EVPN RP Rendezvous Point

Network Underlay
AnyCast RP
ip pim rp-address 10.254.254.1 group-list 239.1.1.0/25
ip pim anycast-rp 10.254.254.1 10.2.0.1
ip pim anycast-rp 10.254.254.1 10.2.0.2
RR RP RR RP

Spine Spine Spine Spine

ip pim rp-address 10.254.254.1 group-list 239.1.1.0/25

VRF-RED VRF-GREEN

V V V V
Leaf Leaf Leaf Leaf

Container
Virtualized
Container
Physical Virtualized
Container
Host
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 RR Route Reflector

VXLAN BGP EVPN RP Rendezvous Point

Network Underlay
interface Ethernet1/2
description connected-to-leaf-1-Ethernet1/1
no switchport
mtu 9216
ip address 10.4.0.5/30 RR RP RR RP
ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode Spine Spine Spine Spine
no shutdown

interface loopback0
description Routing loopback interface
ip address 10.2.0.7/32
ip router ospf UNDERLAY area 0.0.0.0 VRF-RED VRF-GREEN
ip pim sparse-mode

V V V V
router ospf UNDERLAY Leaf Leaf Leaf Leaf
router-id 10.2.0.7

interface Ethernet1/2
description connected-to-spine-1-Ethernet1/1
no switchport
mtu 9216
ip address 10.4.0.6/30
ip ospf network point-to-point Container
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode Virtualized
Container
no shutdown Physical Virtualized
Container
Host
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 RR Route Reflector

VXLAN BGP EVPN RP Rendezvous Point

L2 VNI L3 VNI
Virtual Networks and Overlay Interface vlan 10 vlan 1000
vn-segment 5010 vn-segment 9999
vlan 20 vlan 2000
interface Vlan10 vn-segment 5020 vn-segment 9998
no shutdown
vrf member VRF-RED
RR RP RR RP
no ip redirects
ip address 10.0.0.1/24 tag 12345
no ipv6 redirects Spine Spine Spine Spine
fabric forwarding mode anycast-gateway

interface Vlan20
no shutdown
vrf member VRF-GREEN
no ip redirects
ip address 20.0.0.1/24 tag 12345 VRF-RED VRF-GREEN
no ipv6 redirects
fabric forwarding mode anycast-gateway

V V V V
Leaf Leaf Leaf Leaf
interface nve1
no shutdown
host-reachability protocol bgp
advertise virtual-rmac vrf context VRF-RED IP VRF vrf context VRF-GREEN IP VRF
vni 9999 vni 9998
source-interface loopback1
rd auto rd auto
member vni 5010 Map L2VNI to address-family ipv4 unicast address-family ipv4 unicast
mcast-group 239.1.1.1
member vni 5020
NVE route-target both auto route-target both auto
mcast-group 239.1.1.1 route-target both auto evpn Container
route-target both auto evpn
evpn evpn
member vni 9999 associate-vrf Virtualized
vni 5010 l2 MAC VRF vni 5020 l2 MAC VRF
Container
rd Physical
member vni 9998 associate-vrf
auto rd auto
Virtualized Container
Host both
route-target auto route-target both auto

Associate L3VNI to NVE #CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 RR Route Reflector
VXLAN BGP EVPN RP Rendezvous Point
BGP on Leaf and Spine
router bgp 65010 Leaf BGP Config
router-id 10.2.0.7
address-family l2vpn evpn
Spine BGP Config
advertise-pip router bgp 65010
router-id 10.2.0.2
neighbor 10.2.0.1
remote-as 65010
RR RP RR RP neighbor 10.2.0.3
update-source loopback0 remote-as 65010
address-family l2vpn evpn update-source loopback0
send-community Spine Spine Spine Spine address-family l2vpn evpn
send-community extended send-community
neighbor 10.2.0.2 send-community extended
remote-as 65010 route-reflector-client
update-source loopback0 neighbor 10.2.0.4
address-family l2vpn evpn remote-as 65010
send-community update-source loopback0
send-community extended address-family l2vpn evpn
vrf VRF-RED VRF-RED VRF-GREEN send-community
address-family ipv4 unicast send-community extended
advertise l2vpn evpn route-reflector-client
redistribute direct route-map fabric-rmap-redist-subnet neighbor 10.2.0.5
V V V V remote-as 65010
maximum-paths ibgp 2
address-family ipv6 unicast
Leaf Leaf Leaf Leaf update-source loopback0
advertise l2vpn evpn address-family l2vpn evpn
redistribute direct route-map fabric-rmap-redist-subnet send-community
maximum-paths ibgp 2 send-community extended
vrf VRF-GREEN route-reflector-client
address-family ipv4 unicast
advertise l2vpn evpn
redistribute direct route-map fabric-rmap-redist-subnet
maximum-paths ibgp 2 Container
address-family ipv6 unicast
advertise l2vpn evpn Virtualized

Physical
redistribute direct route-map fabric-rmap-redist-subnet
Virtualized
Host
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
 RR Route Reflector

VXLAN BGP EVPN RP Rendezvous Point

Distributed Anycast Gateway VLAN to VNI Mapping

vlan 20
vn-segment 5020

Anycast Gateway MAC Address- Identically configured on all VTEPs

RR RP forwarding anycast-gateway-mac
fabric RR RP 0002.0002.0002

The same anycast gateway Distributed IP Anycast Gateway (SVI) - Identically configured on all VTEPs
Spine Spineinterface Vlan20 Spine Spine
Virtual IP Address and MAC no shutdown
vrf member VRF-GREEN
address are configured on all no ip redirects
VTEPs in the VNI ip address 20.0.0.1/24 tag 12345
no ipv6 redirects
fabric forwarding mode anycast-gateway

VRF-
SVI VRF-RED SVI
GW IP
SVI SVI
GW IP
GW GW GREEN
GW IP
GW
GW IP
GW
MAC MAC MAC MAC

V V V V
Leaf Leaf Leaf Leaf

Container
Virtualized

Physical Virtualized
Host
#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
EVPN Peer
and Host Discovery
L2VNI
 VXLAN BGP EVPN
Host and Subnet Distribution
RR Route Reflector
• Host Route Distribution
RR
decoupled from the
RR
Underlay protocol
Spine Spine Spine Spine

• Use Multi-protocol-BGP
Overlay (MP-BGP) on the leaf nodes
to distribute internal
Host/Subnet Routes and
V
external reachability
V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4 information

• Route-Reflectors (RR)
deployed for scaling
purposes

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
 VXLAN BGP EVPN
Host and Subnet Routes
RR Route Reflector
• BGP EVPN NLRI
(Network Layer Reachability Information)
RR RR BGP Update Format
Spine Spine Spine Spine
• Host MAC (Route Type 2)
• MAC only, Single VNI, Single

Overlay Route Target

• Host MAC + IP (Route Type 2)

• MAC and IP, Two VNI, Two
V V V V Route Targets, Router MAC
Leaf 1 Leaf 2 Leaf 3 Leaf 4

• Internal and External Subnet

Prefixes (Route Type 5)
• IP Subnet Prefix, Single VNI,
Single Route Target

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
EVPN Control Plane
Host Advertisements

RR RR Overlay (VXLAN)
Spine Spine Spine Spine

Overlay
EVPN RT 2
Update
V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 - RR Overlay
- (VXLAN)
10.200.200.101 0

Spine Spine Spine Spine

Overlay
EVPN RT 2
Update
V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 - RR Overlay
- (VXLAN)
10.200.200.101 0

Spine Spine Spine Spine

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 - RR Overlay
- (VXLAN)
10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
- -
Spine10.200.200.102 0

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 - RR Overlay
- (VXLAN)
10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
- -
Spine10.200.200.102 0

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 - RR Overlay
- (VXLAN)
10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
- -
Spine10.200.200.102 0

2 0000.3002.2101 / 48 3002, 65500:3001 - - 10.200.200.104 0

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 - RR Overlay
- (VXLAN)
10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
- -
Spine10.200.200.102 0

2 0000.3002.2101 / 48 3002, 65500:3001 - - 10.200.200.104 0

Overlay • Host MAC (Route Type 2)

• MAC
• MPLS Label1 (L2VNI)
• Route Target for MAC-VRF

V V V • MAC Attributes are Mandatory

V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
EVPN Peer
and Host Discovery
L3VNI
EVPN Control Plane
Host Advertisements

RR RR Overlay (VXLAN)
Spine Spine Spine Spine

Overlay
EVPN RT 2
Update
V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 RR
192.168.10.101/32 Overlay (VXLAN)
5000, 65500:5000 10.200.200.101 0

Spine Spine Spine Spine

Overlay
EVPN RT 2
Update
V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 RR
192.168.10.101/32 Overlay (VXLAN)
5000, 65500:5000 10.200.200.101 0

Spine Spine Spine Spine

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 RR
192.168.10.101/32 Overlay (VXLAN)
5000, 65500:5000 10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
192.168.10.102/32 5000, 65500:5000
Spine10.200.200.102 0

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 RR
192.168.10.101/32 Overlay (VXLAN)
5000, 65500:5000 10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
192.168.10.102/32 5000, 65500:5000
Spine10.200.200.102 0

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 RR
192.168.10.101/32 Overlay (VXLAN)
5000, 65500:5000 10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
192.168.10.102/32 5000, 65500:5000
Spine10.200.200.102 0

2 0000.3002.2101 / 48 3002, 65500:3001 192.168.20.101/32 5000, 65500:5000 10.200.200.104 0

Overlay
EVPN RT 2
Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
EVPN Control Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq

2 0000.3001.1101 / 48 RR
3001, 65500:3001 RR
192.168.10.101/32 Overlay (VXLAN)
5000, 65500:5000 10.200.200.101 0

2
Spine
0000.3001.1102 / 48
Spine
3001, 65500:3001
Spine
192.168.10.102/32 5000, 65500:5000
Spine10.200.200.102 0

2 0000.3002.2101 / 48 3002, 65500:3001 192.168.20.101/32 5000, 65500:5000 10.200.200.104 0

Overlay • Host MAC+IP (Route Type 2)

• MAC and IP
• MPLS Label1 (L2VNI)
EVPN RT 2 • Route Target for MAC-
Update
VRF
V V V V • MPLS Label2 (L3VNI)
Leaf 1 Leaf 2 Leaf 3 Leaf 4 • Route Target for IP-VRF
• Router MAC

• IP Attributes are Optional

• Populated through ARP / ND

Host A Host B Host D

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3001.1101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
EVPN Control Plane
Subnet Advertisements

RR RR Overlay (VXLAN)
Spine Spine Spine Spine

Overlay
EVPN RT 5 EVPN RT 5
EVPN RT 5 Update
Update Update

V V V V
Leaf 1 Leaf 2 Leaf 3 Leaf 4

Subnet A Subnet A Subnet B

192.168.10.0/24 192.168.10.0/24 192.168.20.0/24

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
EVPN Control Plane
Subnet Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq

5 192.168.10.0/24
RR 5000, 65500:5000 RR Overlay (VXLAN)
10.200.200.101 0

5 192.168.10.0/24 5000, 65500:5000 10.200.200.102 0

Spine Spine Spine Spine
5 192.168.20.0/24 5000, 65500:5000 10.200.200.104 0

Overlay • Internal and External Subnet

Prefixes (Route Type 5)
EVPN RT 2 EVPN RT 5 • IP Prefix
EVPN RT 5 Update
Update Update • MPLS Label (L3VNI)
V V V V • Route Target for IP-VRF
Leaf 1 Leaf 2 Leaf 3 Leaf 4 • Router MAC

• Populated through External Routing

Protocol

Subnet A Subnet A Subnet B

192.168.10.0/24 192.168.10.0/24 192.168.20.0/24

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Packet Walk
Host
Communication in
Same VLAN/VNI
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
V
V
VTEP1
NVE 10.200.200.1
VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001

Host A wants to communicate with Host B

Host A Host Y
Host B
192.168.1.101 192.168.2.102
192.168.1.102

MAC: 0000.3000.1101 MAC: 0000.3000.1102

IP: 192.168.1.101 IP: 192.168.1.102

#CiscoLiveAPJC © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
V
V
VTEP1
NVE 10.200.200.1
VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001
SMAC: 0000.3000.1101
DMAC: 0000.3000.1102

VLAN: 10

SIP: 192.168.1.101
DIP: 192.168.1.102

• Host A generates data traffic with SMAC of 0000.3000.1101 and

Host A
source IP of 192.168.1.101
192.168.1.101
1 Host B
192.168.1.102
Host Y
192.168.2.102

MAC: 0000.3000.1101 • The destination information is set to that of Host B, which is

MAC: 0000.3000.1102
IP: 192.168.1.101 MAC 0000.3000.11102 and IP of 192.168.1.102 IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
• The packet is received at VTEP 1 V
V
VTEP1 2
2• Destination lookup is performed based on VLAN 10 mapped to VNI VTEP2
NVE 10.200.200.2
NVE 10.200.200.1 30001 and 0000.3000.1102
MAC
SVI 10:
IP
SVI 20:
L2VNI Next-Hop Overlay (VXLAN)
SVI 10: SVI 20:
192.168.1. 192.168.1.101
0000.3000.1101 192.168.2.
30001 Local 192.168.1.1 192.168.2.1
1 1
0000.3000.1102 192.168.1.102 30001 10.200.200.2

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001
SMAC: 0000.3000.1101
DMAC: 0000.3000.1102

VLAN: 10

SIP: 192.168.1.101
DIP: 192.168.1.102

Host A
192.168.1.101 Host B Host Y
192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 MAC: 0000.3000.1102

IP: 192.168.1.101 IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
V
V
VTEP1
NVE 10.200.200.1
VTEP2
NVE 10.200.200.2
MAC
SVI 10:
IP
SVI 20:
L2VNI Next-Hop Overlay (VXLAN)
192.168.1. 192.168.1.101 SVI 10: SVI 20:
0000.3000.1101 192.168.2.
30001 Local 192.168.1.1 192.168.2.1
1 1
0000.3000.1102 192.168.1.102 30001 10.200.200.2

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001
3• VXLAN Encapsulation is performed with VNI 30001
• Packet is transported to VTEP2

Host A
192.168.1.101 Host B Host Y
192.168.1.102 192.168.2.102
MAC: 0000.3000.1101
MAC: 0000.3000.1102
IP: 192.168.1.101
IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 4 Underlay Network Routes the packet based on the outer IP header
V
0200.0ade.de02
V
VTEP1
NVE 10.200.200.1
VTEP2
NVE 10.200.200.2
MAC
SVI 10:
IP
SVI 20:
L2VNI Next-Hop Overlay (VXLAN)
192.168.1. 192.168.1.101 SVI 10: SVI 20:
0000.3000.1101 192.168.2.
30001 Local 192.168.1.1 192.168.2.1
1 1
0000.3000.1102 192.168.1.102 30001 10.200.200.2

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001

Host A
192.168.1.101 Host B Host Y
192.168.1.102 192.168.2.102
MAC: 0000.3000.1101
MAC: 0000.3000.1102
IP: 192.168.1.101
IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
V
V
VTEP1
NVE 10.200.200.1
5 VTEP2 Receives VXLAN packet VTEP2
NVE 10.200.200.2
MAC
SVI 10:
IP
SVI 20:
L2VNI Next-Hop Overlay (VXLAN)
192.168.1. 192.168.1.101 SVI 10: SVI 20:
0000.3000.1101 192.168.2.
30001 Local 192.168.1.1 192.168.2.1
1 1
0000.3000.1102 192.168.1.102 30001 10.200.200.2

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001

Host A
192.168.1.101 Host B Host Y
192.168.1.102 192.168.2.102
MAC: 0000.3000.1101
MAC: 0000.3000.1102
IP: 192.168.1.101
IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
V
V
VTEP1
NVE 10.200.200.1
6 VTEP2 performs L2 Lookup based on L2VNI 3001 VTEP2
NVE 10.200.200.2
MAC
SVI 10:
IP
SVI 20:
L2VNI Next-Hop Overlay (VXLAN) MAC IP
SVI 10:
L2VNI
SVI 20:
Next-Hop
192.168.1. 192.168.1.101
0000.3000.1101 192.168.2.
30001 Local 0000.3000.1102 192.168.1.
192.168.1.102 192.168.2.
30001 Local
1 1 1 1
0000.3000.1102 192.168.1.102 30001 10.200.200.2 0000.3000.1101 192.168.2.101 30002 10.200.200.1

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001

Host A
192.168.1.101 Host B Host Y
192.168.1.102 192.168.2.102
MAC: 0000.3000.1101
MAC: 0000.3000.1102
IP: 192.168.1.101
IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
 Packet Walk
ARP Cache populated

Communication between hosts in the same VLAN/VNI

0200.0ade.de01 0200.0ade.de02
V
V
VTEP1
NVE 10.200.200.1
VTEP2
NVE 10.200.200.2
MAC
SVI 10:
IP
SVI 20:
L2VNI Next-Hop Overlay (VXLAN) MAC IP
SVI 10:
L2VNI
SVI 20:
Next-Hop
192.168.1. 192.168.1.101
0000.3000.1101 192.168.2.
30001 Local 0000.3000.1102 192.168.1.
192.168.1.102 192.168.2.
30001 Local
1 1 1 1
0000.3000.1102 192.168.1.102 30001 10.200.200.2 0000.3000.1101 192.168.2.101 30002 10.200.200.1

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001
7
• VXLAN Packet is decapsulated
• Local mapping of VNI 30001 to VLAN 10
• Ethernet Frame sent to Host B

SMAC: 0000.3000.1101
DMAC: 0000.3000.1102

VLAN: 10
Host A
Host Y
192.168.1.101 SIP: 192.168.1.101 Host B
192.168.2.102
192.168.1.102
DIP: 192.168.1.102
MAC: 0000.3000.1101
MAC: 0000.3000.1102
IP: 192.168.1.101
IP: 192.168.1.102

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Packet Walk
Host Communication
in Different
VLAN/VNI
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A V
V
VTEP1
NVE 10.200.200.1
VNI 50001 VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20 VLAN 20
VNI 30002
VLAN 10 VLAN 10
VNI 30001

VRF-A Host A wants to communicate with Host Y VRF-A

Host A Host X Host B Host Y

192.168.1.101 192.168.2.101 192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 MAC: 0000.3000.2102

IP: 192.168.1.101 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A
V
VTEP1
NVE 10.200.200.1
VNI 50001
V
VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20 VLAN 20
VNI 30002
VLANfor10 VLAN 10
ARP Request Anycast GW VTEP1
SMAC:0000.3000.1101 VNI 30001
DMAC: Anycast GW MAC
SIP:192.168.1.101
DIP: 192.168.2.102
VRF-A VRF-A
SMAC: 0000.3000.1101
DMAC: Anycast GW

VLAN: 10
1• Host A generates data traffic with SMAC of 0000.3000.1101 and SIP:192.168.1.101
SIP: 192.168.1.101
DIP: 192.168.2.102 • The destination information is set to DMAC: Anycast GW, and DIP: 192.168.2.102

Host A Host X Host B Host Y

192.168.1.101 192.168.2.101 192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 MAC: 0000.3000.2102

IP: 192.168.1.101 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A V
V
VTEP1
NVE 10.200.200.1
VNI 50001 VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20
VNI 30002 VLAN 20

VLANfor10 VLAN 10
ARP Request Anycast GW VTEP1
SMAC:0000.3000.1101 VNI 30001
DMAC: Anycast GW MAC 2 VTEP1 performs L3 lookup in Routing Table due to DMAC: Anycast GW MAC
SIP:192.168.1.101 2
DIP: 192.168.2.102
VRF-A EVPN Route-Type MAC, IP L2VNI L3VNI Next Hop
VRF-A
2 0000.3000.1101, 192.168.1.101 VNI 30001 - 10.200.200.1

2 0000.3000.1102, 192.168.1.102 VNI 30001 - 10.200.200.2

2 0000.3000.2101, 192.168.2.101 VNI 30002 - 10.200.200.1

2 0000.3000.2102, 192.168.2.102 VNI 30002 - 10.200.200.2

5 IP: 192.168.1.0/24 - VNI 50001 10.200.200.1

10.200.200.2
Host A Host X Host B Host Y
192.168.1.101 192.168.2.101 192.168.1.102 192.168.2.102
5 IP: 192.168.2.0/24 - VNI 50001 10.200.200.1 MAC: 0000.3000.2102
MAC: 0000.3000.1101 10.200.200.2
IP: 192.168.1.101 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A V
V
VTEP1
NVE 10.200.200.1
VNI 50001 VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20
VNI 30002 VLAN 20

• 3 VTEP1 Rewrites Inner MAC-address

• VTEP1 encapsulates the packet in VXLAN
VLAN 10 VLAN 10
• VNI 30001
Sends to VTEP2 across the Overlay network using the L3VNI 50001

VRF-A VRF-A

Host A Host X Host B Host Y

192.168.1.101 192.168.2.101 192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 MAC: 0000.3000.2102

IP: 192.168.1.101 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01
4 Underlay Network Routes the packet based on the outer IP header 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A
V
VTEP1
NVE 10.200.200.1
VNI 50001
V
VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20 VLAN 20
VNI 30002
VLAN 10 VLAN 10
VNI 30001

VRF-A VRF-A

Host A Host X Host B Host Y

192.168.1.101 192.168.2.101 192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 MAC: 0000.3000.2102

IP: 192.168.1.101 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A
V
VTEP1
NVE 10.200.200.1
VNI 50001
V
VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20 VLAN 20
VNI 30002
5 • VTEP2 receives the VXLAN packet based on DMAC
VLAN 10 being its own VLAN 10
VNI• 30001
Performs a routing lookup on L3VNI 50001
• Determines packet is mapped to VRF-A

VRF-A EVPN Route-Type MAC, IP L2VNI L3VNI Next Hop

VRF-A
2 0000.3000.1101, 192.168.1.101 VNI 30001 VNI 50001 10.200.200.1

2 0000.3000.1102, 192.168.1.102 VNI 30001 VNI 50001 10.200.200.2

2 0000.3000.2101, 192.168.2.101 VNI 30002 VNI 50001 10.200.200.1

2 0000.3000.2102, 192.168.2.102 VNI 30002 VNI 50001 10.200.200.2

5 IP: 192.168.1.0/24 - VNI 50001 10.200.200.1

Host A Host X Host B Host Y
192.168.1.101 192.168.2.101
10.200.200.2 192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 5 IP: 192.168.2.0/24 - VNI 50001 10.200.200.1

MAC: 0000.3000.2102
IP: 192.168.1.101 10.200.200.2 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
 Packet Walk
Communication between hosts in different VLAN/VNI (L3)
0200.0ade.de01 0200.0ade.de02
Tenant: VRF-A Tenant: VRF-A V
V
VTEP1
NVE 10.200.200.1
VNI 50001 VTEP2
NVE 10.200.200.2

Overlay (VXLAN)
SVI 10: SVI 20: SVI 10: SVI 20:
192.168.1.1 192.168.2.1 192.168.1.1 192.168.2.1

VLAN 20
VNI 30002 VLAN 20

VLAN 10 VLAN 10
VNI 30001

VRF-A VRF-A

6
• VTEP2 decapsulates the VXLAN packet
• Rewrites MAC and sends Ethernet Frame to Host Y

Host A Host X Host B Host Y

192.168.1.101 192.168.2.101 192.168.1.102 192.168.2.102

MAC: 0000.3000.1101 MAC: 0000.3000.2102

IP: 192.168.1.101 IP: 192.168.2.102
VLAN 10 VLAN 20
VNI 30001 VNI 30002

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
EVPN Host
Mobility
 EVPN Control Plane - Host Mobility
NLRI:
• Host MAC: 0000.3000.1101
• Host IP: 192.168.1.101
• VNI: 5000 Spine Spine
• Next-Hop: VTEP-1

Ext. Community
• Encapsulation: VXLAN
• Cost/Sequence:0

Leaf-1
VTEP1
MAC IP VNI Next-Hop Encap Seq

0000.3000.1101 192.168.1.101 5000 VTEP-1 VXLAN 0

Leaf-2 Leaf-3 Leaf-4
VTEP2 VTEP3 VTEP4 Leaf-20
VTEP20
Host A
MAC: 0000.3000.1101
IP: 192.168.1.101
VNI 5000

1. VTEP1 detects Host A and advertises an EVPN Route for Host A with seq#0

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
 EVPN Control Plane - Host Mobility
NLRI:
• Host MAC: 0000.3000.1101
• Host IP: 192.168.1.101
• VNI: 5000 Spine Spine
• Next-Hop: VTEP-1

Ext. Community
• Encapsulation: VXLAN
• Cost/Sequence:0

Leaf-1
VTEP1

Leaf-2 Leaf-3 Leaf-4

VTEP2 VTEP3 VTEP4 Leaf-20
VTEP20

1. VTEP1 detects Host A and advertises an EVPN Route for Host A with seq#0
Host A
2. Host A moves behind VTEP4
MAC: 0000.3000.1101
IP: 192.168.1.101
VNI 5000

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
 EVPN Control Plane - Host Mobility
NLRI:
• Host MAC: 0000.3000.1101 NLRI:
• Host IP: 192.168.1.101 • Host MAC: 0000.3000.1101
• VNI: 5000 • Host IP: 192.168.1.101
Spine • VNI: 5000Spine
• Next-Hop: VTEP-1
• Next-Hop: VTEP-4
Ext. Community
• Encapsulation: VXLAN Ext. Community
• Cost/Sequence:0 • Encapsulation: VXLAN
• Cost/Sequence:1

Leaf-1
VTEP1 MAC IP VNI Next-Hop Encap Seq

0000.3000.1101 192.168.1.101 5000 VTEP4 VXLAN 1

Leaf-2 Leaf-3 Leaf-4

VTEP2 VTEP3 VTEP4 Leaf-20
VTEP20

1. VTEP1 detects Host A and advertises an EVPN Route for Host A with seq#0
2. Host A moves behind VTEP4 Host A
3. VTEP4 detects Host A and advertises an EVPN Route for Host A with seq#1 MAC: 0000.3000.1101
IP: 192.168.1.101
VNI 5000

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
Conclusion
Control-Plane Comparison
Flood&Learn (RFC7348) EVPN - Ethernet VPN (RFC8365)
• Ethernet over IP • A Better Ethernet over IP
• No Spanning-Tree (terminates at NVE) • No Spanning-Tree (terminates at NVE)
• Endpoint Learning is based on Flood and • Endpoint Learning is based on BGP
Learn (it’s in the name) exchange (EVPN uses BGP)
• Requires Extra Work for Routing • Provides Integrated Routing & Bridging
• FHRP for Default Gateway (IRB)
• Over-the-Top VRF-lite for Prefix Routing (or use • Distributed Anycast Gateway for Default Gateway
the Underlay?!) • Uses a Layer-3 VPN approach like MPLS L3VPN
• And there is much more in EVPN!

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
Conclusion
• Data Center Network Challenges and Evolution

• Virtual Private Networks and Overlay Taxonomy

• VXLAN enables scalable Data Center Networks

• BGP EVPN as a better Control-Plane

• Provides a robust control plane enabling multi-tenancy, VM mobility, and
optimizes forwarding

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
Recommended Reading
• Building Data Centers with VXLAN BGP EVPN: A Cisco NX-OS Perspective
• Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release
10.4(x)
• Migrating Classic Ethernet Environments to VXLAN BGP EVPN
• Migrating Cisco FabricPath Environments to VXLAN BGP EVPN White Paper
• VXLAN EVPN Multi-Site Design and Deployment White Paper
• VXLAN Design with Cisco Nexus 9300 Platform Switches
• VXLAN Network with MP-BGP EVPN Control Plane Design Guide

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
Recommended CiscoLive Sessions
• VXLAN BGP EVPN Multi-Site [BRKDCN-2913]
• VXLAN EVPN: A Deep Dive into Packet Forwarding [BRKDCN-3966]
• Next Generation VXLAN Data Centre Architecture, Deployment, and
Operation [TECDCN-2101]
• Architecting L4-L7 Network Services in a Multi-tenant Data Centre
with VXLAN EVPN [BRKDCN-2974]
• Design, Automate, and Manage VXLAN BGP EVPN Multi-Site with
NDFC [BRKDCN-2128]

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
Session Surveys
We would love to know your feedback on this session!
• Complete a minimum of four session surveys and the overall event surveys to claim
a Cisco Live T-Shirt

#CiscoLiveAPJC BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
 • Visit the Cisco Showcase for
related demos

• Book your one-on-one

Meet the Expert meeting

• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs

your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

BRKDCN-1621 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
Thank you

#CiscoLiveAPJC
#CiscoLiveAPJC
#CiscoLiveAPJC