BRKDCT 2949

BRKDCT-2949
Building Data Center

Networks with VXLAN EVPN
Overlays – Part I
Lukas Krattiger, Principal Engineer

Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKDCT-2949
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objective
• A short Overview on Data Center Fabric
• A close look at Single Fabric Overlay and Underlay
• Details on Single Fabric Control- & Data-Plane
• Multi-Tenancy in VXLAN BGP EVPN environments
• First-Hop Gateway with Distributed Anycast Gateway
• Multi-Homing with Virtual Port-Channel (VPC) for VXLAN
BRKDCT-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Introduction to Data Center Fabric

• Leaf, Spine, Super-Spine (CLOS)
• Overlay
• Underlay
• VXLAN with BGP EVPN

• Control & Data Plane
• Multi-Tenancy
• Distributed Anycast Gateway
• VPC
• A Deployment Story
Introduction to Data Center Fabrics
Data Center “Fabric” Journey (Standalone)
Layer-3 HSRP HSRP
Layer-2
Spanning-Tree
Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2
Baremet al Hypervisor Hypervisor Hypervisor Baremet al Hypervisor Baremet al Baremet al Hypervisor Hypervisor
BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
The Leaf / Spine Topology (Clos* Network)
Spine Spine Spine Spine
• Wide ECMP: Unicast or Multicast

• Uniform Reachability
• Deterministic Latency Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• High Redundancy
• On Node or Link Failure
*Clos, Charles (1953) "A study of non-blocking switching networks" BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
A Scale Out Architecture
More Spine – More Bandwidth – More Resiliency

• Leaf Spine Spine Spine Spine
• Smallest Operational Entity
• Spines
• Wide vs. Big
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Uplinks
• Symmetric to all Spines or Pods
More Leaf – More Ports – More Capacity
• SAYG: Scale as You Grow
The Super-Spine
SuperSpine
SuperSpine SuperSpine
Spine Spine Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
POD 1 POD 2
The Super-Spine
SuperSpine
• Scale Out
• Not Limited to Port Density

• Simpler Capacity Planning
• Beyond a single Server Room

• Allows Interconnecting Pods
Spine Spine Spine Spine Spine Spine Spine Spine
• Retains Intra-Pod Topology with

Flexible Inter-Pod Connectivity
POD 1 POD 2
Data Center Fabric Properties
• Any Subnet, Anywhere, Rapidly

• Any Network on Any Leaf
• Reduced Failure Domain

• Any Default Gateway on Any Leaf
- Distributed
• Extensible Scale and Resiliency
Overlay Based Data Center: Fabrics
• Mobility
• Segmentation
Overlay • Scale
VTEP VTEP VTEP VTEP VTEP VTEP VTEP

• Automated and Programmable
• Abstracted Consumption Model
• Layer-2 and Layer-3 Service
• Physical and Virtual Workloads
Overlay Based Data Center: Edge Devices
Network Overlays Host Overlays
Overlay Overlay
VTEP VTEP VTEP VTEP - - - -
Hybrid Overlays
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal Baremetal Hypervisor Hypervisor Hypervisor Hypervisor
• Router/Switch End-Points
• Virtual End-Points only
• Protocols for Resiliency/Loops Overlay • Single Admin Domain
• Traditional VPNs
• VXLAN, NVGRE, STT
• VXLAN, OTV, VPLS, LISP, FP - - VTEP VTEP
VTEP VTEP
Hypervisor Hypervisor Baremetal Baremetal
• Physical and Virtual

• Resiliency and Scale
• Cross-Organizations/Federation
• Open Standards
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
Overlay Taxonomy - Underlay
Layer-3
Interface Spine Spine Spine Spine
Peering
Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Virtual
Server Physical
Server
Overlay Taxonomy - Overlay
Tunnel Encapsulation
Spine
(VNI Namespace)
Spine Spine Spine
Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
LAN
Segment
Virtual
Server Physical VTEP: VXLAN Tunnel End-Point
Server VNI/VNID: VXLAN Network Identifier
Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3
Data-Plane
Control-Plane • Overlay Layer-2/Layer-3 Unicast Traffic
• Peer-Discovery • Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution Multicast traffic (BUM traffic)
• Local Learning forwarding
• Remote Learning • Ingress Replication (Unicast)
• Multicast
VXLAN Evolves as the Control Plane Evolves!
Back Then
Yet Another Encapsulation
 Flood & Learn (Multicast-based)
 Data-Plane only 4 Years ago
VXLAN for the Data Center – Intra-DC
 Control-Plane
 Active VTEP Discovery
 Multicast and Unicast
A single Fabric with Overlay
• A Single Overlay Domain

• End-to-End Encapsulation
• Closest to the Source
• Closest to the Destination
Overlay • External Connectivity

• @ Leaf = Border Leaf
• @ Spine = Border Spine
POD 1
What is the Elephant in the Room?
The Super-Spine and the Overlay (Multi-POD)
SuperSpine
Spine Overlay
Spine Spine Spine Spine Overlay
Spine Spine Spine
POD 1 POD 2
SuperSpine
• Still, a Single Overlay Domain
• End-to-End Encapsulation
Spine Overlay
Spine Spine Spine • External Connectivity
OverlaySpine Spine Spine Spine

• @ Spine = Border Spine
• @ Super-Spine or Connected to
Super-Spine
POD 1 POD 2
Attributes of Multi-POD in VXLAN BGP EVPN
SuperSpine
• Underlay
•
Nicely Structured and Tiered Topologies

• Allows Efficient Scale-Out
• More End-Points = More Leaf
• More Bandwidth, Resilience or Capacity = More Spine or Tiers
• Different Control-Plane Instances (BGP AS)
Spine Overlay
Spine Spine Spine
• Overlay
• End-to-End Encapsulation, Flat, No Hierarchy
• Single Control-Plane ”reach” – all in one ”kitchen sink”
POD 1 POD 2
SuperSpine
• Scale-Out Model to Build a Large

Intra-DC Network?
• Data Center Interconnect (DCI)?
Spine Overlay
Spine Spine Spine
• Domain Normalization
(Coexistence and/or Migration)?
POD 1 POD 2
VXLAN Evolves as the Control Plane Evolves!
Back Then
Yet Another Encapsulation
 Flood & Learn (Multicast-based)
 Data-Plane only 4 Years ago
VXLAN for the Data Center – Intra-DC
 Control-Plane
 Active VTEP Discovery
Today
 Multicast and Unicast
VXLAN for DCI – Inter-DC
 Multi-Site
 Control- & Data-Plane Separation
 Failure Domain Isolation
Use-Cases for ”DCI” Connectivity
Scale-Out Model to Build a Large Intra-
DC Network
Data Center Interconnect (DCI)
Domain Normalization
(Coexistence and/or Migration)
VXLAN for Interconnecting Networks
VXLAN Multi-Pod VXLAN Multi-Fabric
EVPN Control-
Fabric #1 BGP EVPN EVPN Control-
Fabric #2 EVPNFabric
Control-Plane
#1 EVPNFabric
Control-Plane
#2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar
eta eta eta eta em em em em
l l l l etal etal etal etal
DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane
 Single Fabric with End-to-End  Multiple Fabrics – Normalized

Encapsulation through Ethernet
 Build Hierarchy in the Underlay  Multiple Fabrics Interconnect
– Flatten it in the Overlay using DCI (Layer 2 and Layer 3)
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site
EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal
DCI DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane
 Single Fabric with End-to-End  Multiple Fabrics – Normalized  Multiple Fabrics with
Encapsulation through Ethernet Integrated DCI
 Build Hierarchy in the Underlay  Multiple Fabrics Interconnect  Integrated DCI – Scaling
– Flatten it in the Overlay using DCI (Layer 2 and Layer 3) within and between Fabrics
VXLAN Multi-Site – Introducing Overlay Hierarchies
SuperSpine
Border Gateways (BGW) SuperSpine SuperSpine
(Key Functional Components of
VXLAN Multi-Site Architecture) Multi-Site Overlay
VTEP VTEP
Spine Overlay
Spine Spine Spine
Site 1 Site 2
VXLAN Multi-Site – Introducing Overlay Hierarchies
SuperSpine
• Multiple Overlay Domains
• Per-Site Encapsulation
Multi-Site Overlay
VTEP • Exit/Transit via Border Gateway
VTEP
(BGW)
Spine Overlay
Spine Spine Spine
• Multi-Site and/or External

Connectivity
•
@ Spine = Border Spine

• Super-Spine becomes Transit
Site 1 Site 2
VXLAN Multi-Site for Interconnecting Networks
Your Happy Place! 

Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
MTU and Overlays
• Data Center often require Jumbo
MTU
• Most Server NIC support up to
9000 Bytes
• Network Switches support MTU

up to 9216* Bytes
• Accommodates Jumbo MTU plus
Overlay overhead
• Avoid Fragmentation
• Adjust the Transport Network with
appropriate MTU
*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Interface Principles
• Routed Ports and Interfaces

• Layer-3 Interfaces between Leaf Spine Spine Spine Spine
and Spine(no switchport)

• For each Point-2-Point (P2P) Underlay
connection, minimum /31
required Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Alternative, use IP Unnumbered

(/32)
• Use Loopback as Source-

Interface for VTEP (NVE*)
*NVE: Network Virtualization Edge BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
IP Addressing Principles Routing
Identifier Rendezvous
• Prepare a IP Addressing Plan p2p* Links / IP Point
Unnumbered
• Separate Interface functions Spine Spine Spine Spine
through IP Addressing
(Aggregates) Underlay
• Unicast Routing – Routing
Protocol Peering (p2p*) Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Unicast Routing – Routing

Identifier (RID)
Routing
• VTEP and VPC VTEP
Identifier
• Multicast Rendezvous-Point (RP) Loopback
p2p Agg: 10.1.1.0/24

• IPv4 only (today) RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
*p2p: Point-to-Point BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
IP Addressing Principles
interface loopback254
description RP
interface ethernet4/4 ip address 10.254.254.1/32
description p2p-to-Leaf interface loopback0
ip address 10.1.1.2/30 description RID
Spine Spine Spine Spine ip address 10.10.10.201/32
interface ethernet1/49
description p2p-to-Spine
ip address 10.1.1.1/30 Underlay
interface loopback0
description RID
ip address 10.10.10.101/32 interface loopback1
Hypervisor Baremetal description VTEP
Hypervisor Hypervisor Baremetal Hypervisor Baremetal
p2p Agg: 10.1.1.0/24
Baremetal
ip address 10.200.200.101/32
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
Some Math – IP Addressing Principles (P2P)
• Example from depicted Topology

• 4 Spine * 7 Leaf (28 Links) Spine Spine Spine Spine
• 11 Router ID (RID Loopback)

• 7 VTEP (Loopback) Underlay
• 28 Link * 2 (/31) = 56 IP Addresses
• 11 Router ID (RID) = 11 IP Addresses

• 7 VTEP = 7 IP Addresses
• Total: 74 IP Addresses Required
Simplifying the Math – IP Unnumbered
• Example from depicted Topology

• 4 Spine + 7 Leaf (11 Loopback) Spine Spine Spine Spine
• 11 Router ID (RID Loopback)

• 7 VTEP (Loopback) Underlay
• 11 Unnumbered IF = 11 IP Addresses
• 11 Router ID (RID) = 11 IP Addresses

• 7 VTEP = 7 IP Addresses
• Total: 29 IP Addresses Required
Unicast Routing – OSPF and IS-IS
• OSPF – watch your Network Type • IS-IS – what was this CLNS?
• Independent of IP (CLNS)
• Network Type Point-2-Point • Well suited for routed
• Preferred (only LSA type-1) interfaces/ports
• No DR/BDR election • No SPF calculation on Link change;
• Suits well for routed interfaces/ports only if Topology changes
(optimal from a LSA DB perspective) • Fast Re-convergence
• Full SPF calculation on Link Change • Not everyone is familiar with it
Unicast Routing – BGP
• eBGP Underlay Routing –

Service Provider style
• Two Different Models
• Two-AS
• Multi-AS
• BGP is a Distance Vector

Protocol
• actually Path Vector Protocol
• AS* are used to calculate the
Path (AS_Path)
Unicast Routing – eBGP Two-AS Model
All-Spine AS#65500
• eBGP Two-AS, yes it works!
• eBGP peering for Underlay

Underlay • Spine is not a Route-Reflector
(eBGP) – Retain Route-Targets
• Disable BGP AS-Path check
• Next-Hop needs to be
Unchanged
All-Leaf AS#65501
• Underlay is Reachability!
• Advertise your Loopbacks
Unicast Routing – eBGP Multi-AS Model
• eBGP Two-AS, yes it works!
All-Spine AS#65500
• Spine is not a Route-Reflector
(eBGP) – Retain Route-Targets
Underlay • Disable BGP AS-Path check
• Next-Hop needs to be
Unchanged
• Underlay is Reachability!
• Advertise your Loopbacks
• Changes Overlay Routing Policy

• Manually define Route-Targets
Unicast Routing – eBGP Model
Spine Spine Spine Spine • Two different BGP Peering

Underlay • Global IPv4/v6 Address-Family
• Use Physical Interface IP
• eBGP peering for Overlay

• Global EVPN Address-Family
• Use Loopback Interface IP
• BFD not so ok
Unicast Routing – Why two different BGP Peering?
1) Interface Down – BGP Down
BGP Peering
Spine • Point-2-Point
(IPv4/IPv6) Link Fails
BGP Peering
(IPv4/IPv6)
• BGP Peering is teared down
• Lights-Out Event or BFD
Leaf
• Fast reaction
Leaf
to Routing Table
AS#65500
• Underlay Network Converges
• ECMP kicks in if
available/configured
Spine
BGP Peering • IGPsBGP

doPeering
this Automatically
(IPv4/IPv6) (IPv4/IPv6)
• Point-2-Point
BGP Peering
Link Fails
Spine
2) Interface Down – BGP Not Down (EVPN)
• Loopback to Loopback Peering
remains Up
• If Alternate Path available
Leaf • Timers should
Leaf allow Time for
AS#65500 Network Re-Convergence
• No BFD
Spine
• Unchanged Overlay Reachability
• No Mass Delete/Re-Learn
• Underlay Path change only
• Leaf and p2p Interfaces Fail
2) Interface Down – BGP Not Down
Spine
• Either IGP or BGP converges
• Loopback to Loopback Peering

3) Leaf Down – Prefix are Withdrawn (RNH*) remains Up
Leaf • BGP Dead-Timer
Leaf (180s)
AS#65500
• Recursive Next-Hop will trigger
Convergence Event
5 192.168.10.0/24
Spine
• Next-Hop (VTEP) disappeared in
Next-Hop:
Underlay
2 0000.3001.1101
10.200.200.102 • Overlay withdraws Prefixes
2 0000.3001.1101, 192.168.10.101
*RNH: Recursive Next-Hop BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Underlay - Unicast Routing and Overlay
IGP + BGP for
• Generic Concept for Underlay / true Protocol
Overlay Separation Separation
• Use Different Routing Protocol BGP for single
• Use Same Routing Protocol Routing Protocol
Specific to BGP approach
as a Overlay • RNH* for Overlay works with ALL
Control-Protocol Underlay Routing Protocols
• Ensure /32-Reachability for
BGP Knobs can VTEPs
Help here • Other Routes can impact
(Aggregates, Default-Route)
Underlay - Multicast Routing and Rendezvous-Point
• PIM Any-Source-Multicast (ASM) • Bidirectional PIM (Bidir)

• Platform Support • Platform Support
• Nexus 9000 / Nexus 7000 (F3/M3) • Nexus 5600 / Nexus 7000 (F3/M3)
• ASR 1000 / ASR 9000 • ASR 1000 / ASR 9000
• RP Redundancy • RP Redundancy
• PIM Anycast-RP or MSDP • Phantom-RP
• Source-Trees (Unidirectional) • Shared-Trees (Bidirectional)

• 1 Source Tree per VTEP per • 1 Shared Tree per Multicast Group
Multicast Group • Follows Unicast Routing Path
Underlay – PIM ASM with PIM Anycast-RP
RP RP
S,G S,G
S,G S,G
Underlay S,G
PIM ASM – S,G for 5 VTEP
S,G
S=VTEP1 S,G
S=VTEP2 S=VTEP3
S,G
S,G S=VTEP4
S=VTEP7
S,G
Underlay – PIM ASM with PIM Anycast-RP
ip pim anycast-rp 10.254.254.1 10.10.10.201
ip pim anycast-rp 10.254.254.1 10.10.10.202
ip pim rp-address 10.254.254.1

RP RP (Leaf&Spine)
interface loopback254 Spine Spine Spine Spine interface loopback254
description RP description RP
ip address 10.254.254.1/32 ip address 10.254.254.1/32
ip pim sparse-mode ip pim sparse-mode
Underlay
interface loopback0 interface loopback0
description RID description RID
ip address 10.10.10.201/32
Leaf Leaf Leaf Leaf Leaf Leaf ipLeaf address 10.10.10.202/32
Underlay - Multicast Routing and Rendezvous-Point
• The Spine makes a good
Rendezvous-Point (RP)
• Use multiple RP for Redundancy
• Watch your Multicast-Group and
OIF* scale
• VXLAN uses Multicast for BUM

• Broadcast, Unknown Unicast,
Multicast
• 1:1 Multicast-to-VNI mapping
• 1:N Multicast-to-VNI mapping
• Ingress-Replication can be valid

as well
*OIF: Outgoing Interface BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Underlay – Ingress Replication
• A Packet Multiplication
• EVPN assists, VNI Topology
• Various Platform Support

• Ie Nexus 9000 Underlay
• Ingress Replication
•
Host sends 1 Packet to Edge-Device
• Edge-Device Encapsulates 1 Packet
and multiplies it
• Ingress VTEP sends 1 Packet per
Neighbor
VXLAN with BGP EVPN
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
What is … ?
• VXLAN • EVPN
• Standards based Encapsulation • Standards based Control-Plane
• RFC 7348 • RFC 7432
• Uses UDP-Encapsulation • Uses Multiprotocol BGP
• Transport Independent • Uses Various Data-Planes

• Layer-3 Transport (Underlay) • VXLAN (EVPN-Overlay), MPLS,
Provider Backbone (PBB)
• Flexible Namespace
• 24-bit field (VNID) provides ~16M • Many Use-Cases Covered
unique identifier • Bridging, MAC Mobility, First-Hop &
• Allows Segmentations Prefix Routing, Multi-Tenancy (VPN)
Introducing Ethernet VPN (EVPN)
EVPN MP-BGP – RFC 7432
MPLS Provider Backbone Bridges Overlay (NVO3)
(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)
VXLAN and EVPN related RFCs & Drafts (IETF)
ID Title Category
RFC 7348 Virtual Extensible Local Area Network Data Plane
RFC 7432 BGP MPLS based Ethernet VPNs Control Plane
draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane
draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane
draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane
draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane
Multiprotocol BGP (MP-BGP) Primer
• Multiprotocol BGP (MP-BGP)

Spine
• Extension to Border Gateway

Protocol (BGP)
• RFC 4760
Leaf Leaf
AS#65500 • VPN Address-Family
• Allows different types of Address-
Families (i.e. VPNv4/v6, MVPN,
Spine
L2VPN, EVPN)
• Various Information transported
over single BGP Peering
vrf context A vrf context A
rd 10.10.10.101:3 rd 10.10.10.102:8
address-family ipv4 unicast address-family ipv4 unicast
route-target import 65500:5000
route-target export 65500:5000
Spine
• VPNroute-target
Segmentation
route-target
import 65500:5000
for 65500:5000
export Tenant
Routing
• Route Distinguisher (RD)
Leaf • 8-byte field
Leaf
AS#65500
• A Value to make a VPN Prefix
unique
Spine
• RD + VPN Prefix
• [10.10.10.101:5000 + 192.168.10.0/24]
rd auto rd auto
route-target import 65500:5000 Spine route-target import 65500:5000
route-target export 65500:5000 route-target export 65500:5000
• Cisco provides automated Route
Distinguisher derivation
Leaf • Macros uses Type 1 format Leaf
AS#65500 • 4-byte Router ID (RID)
• 4-byte VRF ID (internal number)
• Example of auto derived RD:
Spine
• 10.10.10.101:3
rd auto rd auto
address-family ipv4 unicast • VPNaddress-family
Segmentation forunicast
ipv4 Tenant
route-target export 65500:5000 Routing
route-target export 65500:5000
• Route Target (RT)

• 8-byte field
Leaf Leaf
AS#65500 • A Value to import/export a VPN
Prefix
• Each RD + VPN Prefix have an
Spine
RT
• [10.10.10.101:5000 + 192.168.10.0/24]
• [65500:5000, 65500:5000]
rd auto rd auto
route-target import auto Spine route-target import auto
route-target export auto • Cisco provides automated
route-target export auto Route
Target derivation
• Macros uses following values
•
Leaf Leaf
AS#65500 4-byte Autonomous System
• 4-byte VNI
• Example of auto derived RD:
• Import, Export or Both
Spine
• 65500:5000
rd 10.10.10.101:3 rd 10.10.10.102:8
RD Prefix Next-Hop Route Target

Leaf Leaf
10.10.10.101:3 192.168.10.0/24 10.200.200.101 AS#65500

65500:5000, 65500:5000
10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001
Spine
rd 10.10.10.101:3 rd 10.10.10.102:8

Leaf Leaf
AS#65500
10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000
10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001
Spine
rd 10.10.10.101:3 rd 10.10.10.102:8

Leaf Leaf
AS#65500
10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000
10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001
Spine
rd 10.10.10.101:3 rd 10.10.10.102:8
Leaf Leaf
AS#65500
Next-Hop:
Spine 5 192.168.10.0/24
10.200.200.101
EVPN - Host and Subnet Route Distribution
• Host Route Distribution
RR RR decoupled from the Underlay
protocol
Overlay • Use MultiProtocol-BGP (MP-

BGP) on the Leaf nodes to
Leaf Leaf Leaf Leaf Leaf Leaf Leaf distribute internal Host/Subnet
Routes and external reachability
information
• Route-Reflectors (RR) deployed
for scaling purposes
EVPN Control Plane - Host and Subnet Routes
• BGP EVPN NLRI*

• Host MAC (Route Type 2)
• MAC only, Single VNI, Single
Route Target
Overlay
• Host MAC+IP (Route Type 2)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • MAC and IP, Two VNI, Two Route
Target, Router MAC
• Internal and External Subnet

Prefixes (Route Type 5)
• IP Subnet Prefix, Single VNI,
Single Route Target
*NLRI: Network Layer Reachability Information (BGP Update Format) BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104
Overlay
Baremetal Baremetal Baremetal
Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
*L2VNI: VNI for all Bridging operation (”VLAN-VNI”) BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

• Host MAC (Route Type 2)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 • MAC
10.200.200.107
• MPLS Label1 (L2VNI*)

• Route Target for MAC-VRF
• MAC attributes are Mandatory

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
*L2VNI: VNI for all Bridging operation (”VLAN-VNI”) BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L2VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label1)
10.200.200.101 (metric 3) fromL2VNI
10.10.10.201 (10.10.10.201)
Encap:8
Route Target VXLAN
Origin IGP, MED not set, localpref 100, weight 0
Received label 3001
Extcommunity: RT:65500:3001 ENCAP:8
Originator: 10.10.10.101 Cluster list: 10.10.10.201
Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104
Overlay

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
*L3VNI: VNI for all Routing operation (”VRF-VNI”) BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 • Host

5000, 65500:5000 MAC+IP (Route Type 2)
10.200.200.101
•
MAC and IP
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000
• 10.200.200.104
MPLS Label1 (L2VNI)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 192.168.20.101 /32 5000, 65500:5000• Route Target for MAC-VRF
10.200.200.107
• MPLS Label2 (L3VNI*)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Route Target for IP-VRF
• Router MAC
• IP Attributes are Optional

Baremetal Baremetal Baremetal • Populated through ARP/ND
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
MAC/IP for VRF default, address family L2VPN EVPN
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,
version 4
Paths: (1 available, best #1) IP Address
Length
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked IP Address
Next-Hop L3VNI
IP Address
Path type: internal,L2VNI
path is (MPLS
valid, is best path, no labeled nexthop
Label2)
AS-Path: NONE, path
(MPLSsourced
Label1) internal to AS
10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 VXLAN
Received label 3001 5000
Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
L2VNI L3VNI
Route Target Router MAC
Route Target
Subnet Route Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq.
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

• Internal and External Subnet
Prefixes (Route Type 5)
Overlay • IP Prefix
• MPLS Label (L3VNI)
• Route Target for IP-VRF
•
101010110101
01010101010
Router MAC
• Populated through External

Routing Protocol
Subnet A
192.168.10.0/24
Subnet Route Advertisements
5 192.168.10.0 /24
Spine
5000, 65500:5000
Spine Spine
10.200.200.101
Spine
• IP Prefix Learning
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.104
• via BGP with VRF-Lite
• via LISP on Nexus 7000/7700
5 192.168.20.0 /24
Overlay
5000, 65500:5000 10.200.200.107 • via other routing protocol (static
or dynamic)
• Default: Export of IP Host and IP

101010110101 101010110101
01010101010 01010101010
Prefix Routes advertisements

• Filter and Summarize where
appropriate
Subnet A Subnet A Subnet B
192.168.10.0/24 192.168.10.0/24 192.168.20.0/24
Ethernet Tag
Ethernet Identifier
Segment
V2# show bgp l2vpn evpn 192.168.10.0 (Ethtag)
Identifier (ESI) IP Address
Route Type: IP Address
Length family
IP Prefix for VRF default, address L2VPN EVPN
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Next-Hop
IP Address
Path type: internal, path
L3VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label)
10.200.200.101 (metric 3) fromL3VNI
10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 Router MAC
Route Target VXLAN
Received label 5000
Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Introducing VXLAN
MAC 802.1q IP Payload CRC

Src and Dst
Src, Dst VTEP VTEP IP
and Hop-by- UDP Dst VXLAN
Address Port 4789 VNI
Hop MAC Original Layer-2 Frame
Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN) 20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes

of total overhead
UDP Src Port

Hash of L2/L3/L4
headers of original
Frame
*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
VXLAN Frame Format – MAC in IP Encapsulation
Field Value Bites Total Field Value Bites Total
Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16
(4 Bytes Optional)
8 Bytes
Src. MAC Address Next-Hop MAC Address 48 Destination Port 4789 (UDP) 16
14 Bytes
VLAN Type 0x8100 16 UDP Length 16
VLAN ID Tag 16 Checksum 0x0000 16
Ether Type 0x0800 16
Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC
Field Value Bites Total

Field Value Bites Total
VXLAN Flags RRRRIRRR 8
IP Header Misc. Data 72
8 Bytes
Reserved 24
Protocol 0x11 (UDP) 8
20 Bytes
VNI 16M Possible Segments 24

Header Checksum Various 16
Reserved 8
Source IP Src, VTEP IP 32
Destination IP Dest. VTEP IP 32 BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
No Path Diversity
Spine
• Equal Cost Multi-Pathing (ECMP)
uses Header information to form
Path Diversity
Leaf
• Some Tunnel Protocol provide no
Leaf
101010110101010
10101010
Baremetal
AS#65500 diversity in IP or Protocol Header Baremetal
• As a Result, all Packets travel

the same Path
Spine
• No Path Diversity or Entropy
Introducing VXLAN – Entropy
Spine
• VXLAN provides variable UDP
Source Port in Outer Header
• Hash of the inner Layer-2/Layer-
VTEP VTEP
101010110101010
10101010
Baremetal
AS#65500 3/Layer-4 Headers of the original Baremetal
Ethernet Frame.
• Enables entropy for ECMP Load
Spine
balancing in the Network
Introducing VXLAN – Entropy
Spine
VTEP VTEP
AS#65500
Entropy Spine
happens here
Data-Plane (VXLAN)
Difference between VXLAN (F&L) and VXLAN (EVPN)?
F&L – Flood & Learn • EVPN – BGP EVPN
• Data-Plane Encapsulation • Control-Plane + Data-Plane Solution
• Layer-2 MAC-in-IP Encapsulation • Layer-2 MAC-in-IP Encapsulation with
Reachability Protocol
• Follows Ethernet Semantics –
Learning through Flooding • Follows local/remote learning
through Control-Plane
• No Layer-3 • Learn Local, advertise to Remote
• First-Hop Gateway, Multi-Tenancy
• Integrated Layer-2 and Layer-3
• Uses Multicast for BUM • First-Hop Gateway, Multi-Tenancy
• BUM – Broadcast, Unknown Unicast,
Multicast • Uses Multicast or Ingress
• Some static Ingress Replication (IR) Replication for BUM
possible • Ingress Replication (IR) / Head-End
Replication (HER)
VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)
3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Dst VTEP IP L2VNI Dst MAC Dst IP

10.200.200.101 3001 0000.3001.1101 192.168.10.101
Data-Plane (VXLAN)
Bridging
VXLAN and BGP EVPN – Putting it Together
Extended Community
Router MAC
Control-Plane (BGP EVPN) 0200.0ade.de01
3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Dst VTEP IP L3VNI Router MAC Dst IP

10.200.200.101 5000 0200.0ade.de01 192.168.10.101
Data-Plane (VXLAN)
Routing
Routing and the Router MAC – Ethernet
Router MAC
SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1
interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Routing and the Router MAC – VXLAN
Router MAC
SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1
interface: NVE1 interface: NVE1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Packet Walk – ARP Request
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101
SIP DIP VXLAN Overlay

SMAC DMAC
ARP Request for
192.168.10.102
10.200.200.101 239.0.0.1 3001 0000.3001.1101 FFFF.FFFF.FFFF

ARP Request for 192.168.10.102
SMAC: DMAC:
ARP Request for 192.168.10.102 0000.3001.1101 FFFF.FFFF.FFFF
SMAC: DMAC:
0000.3001.1101 FFFF.FFFF.FFFF

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – ARP Response
Type
Type MAC
MAC/ Length
/ Length L2VNI
L2VNI/ RT
/ RT IPIP/ Length
/ Length L3VNI
L3VNI/ RT
/ RT Next-Hop
Next-Hop Seq.
Seq.
22 0000.3001.1101
0000.3001.1101/ 48
/ 48 3001,
3001,65500:3001
65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
10.200.200.101
SIP DIP VXLAN Overlay

SMAC DMAC
ARP Response for
192.168.10.102
10.200.200.104 10.200.200.101 3001 0000.3001.1102 0000.3001.1101
ARP Response for 192.168.10.102 ARP Response for 192.168.10.102
SMAC: DMAC: SMAC: DMAC:

0000.3001.1102 0000.3001.1101 0000.3001.1102 0000.3001.1101

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Bridging
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102/32 5000, 65500:5000 10.200.200.104
SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.104 3001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102
SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102
Baremetal 192.168.10.101 192.168.10.102 Baremetal Baremetal

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Routing
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP

Router MAC
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Routing (Silent Host)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Routing (Silent Host)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107
2 0000.3002.21o1 / 48 3002, 65500:3002 192.168.20.101 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

ARP Response for 192.168.20.101
SMAC: DMAC:
0000.3002.2101 2020.0000.AAAA
ARP Request for 192.168.20.101
SMAC: DMAC:
SMAC DMAC SIP DIP 2020.0000.AAAA FFFF.FFFF.FFFF
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
What is Multi-Tenancy ?
• Segregation at Layer-2 • Segregation at Layer-3

• VLAN • VRF
• Layer-2 VNI (L2VNI) • Layer-3 VNI (L3VNI)
• VLAN Significance • VRF Significance

• Per-Fabric • Per-Fabric
• Per-Switch • Per-Switch
• Per-Port
Layer-2 Multi-Tenancy – Bridge Domains
• Bridge Domain
• Layer-2 Segment from End-Point
to End-Point
Overlay • Bridge Domains in VXLAN

VNI 3001 (L2VNI)
consists of
• The Ethernet Segment (VLAN)
between Host and Edge Device
• The Hardware Resources within
VLAN 10 the Edge Device
VLAN 100
• The VXLAN Segment (VNI)
between Edge Device and Edge
Host A
VLAN 10
Host B
VLAN 100
Host C
VLAN 20
Device
Layer-3 Multi-Tenancy – Routing Domains
• Routing Domain
• Multiple Subnets sharing the
same Layer-3 forwarding policy
Overlay
VNI 5000 (L3VNI) • Routing Domain in VXLAN
consists of
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • The Routing Domain local to the
Edge Device (VRF)
• The Routing Domain (VPN)
across the Edge Devices
• Multi-Protocol BGP with EVPN
Address-Family
192.168.10.101 192.168.10.102 192.168.20.101
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
Distributed IP Anycast Gateway
• Distributed First-Hop Routing on
192.168.20.1
192.168.10.1 2020.0000.AAAA Edge Device
2020.0000.AAAA
•
All Edge Device share same
Gateway IP and MAC address
Overlay • Pervasive Gateway approach

• Gateway is always active
• No redundancy protocol for hello
or state exchange
• Distributed and smaller state

• Only local End-Points ARP
entries
Distributed IP Anycast Gateway
• Distributed First-Hop Routing on
192.168.20.1
192.168.10.1 2020.0000.AAAA Edge Device
2020.0000.AAAA
•
All Edge Device share same
Gateway IP and MAC address
Overlay • Pervasive Gateway approach

• Gateway is always active
• No redundancy protocol for hello
or state exchange
• Distributed and smaller state

• Only local End-Points ARP
entries
Anycast – One-to-Nearest Association

• Network Addressing and Routing
Methodology
Overlay • Datagrams sent from a single
Sender to the Topologically
Nearest Node
• Group of potential Receivers, all
identified by the same
Destination Address
Packet Walk – Symmetric IRB (A to C)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Symmetric IRB (C to A)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.107 10.200.200.101 5000 0200.0ade.de07 0200.0ade.de01 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101


MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
VPC Gateway Redundancy
• VPC – Virtual Port-Channel
• Multi-Chassis Link Aggregation
• Layer-2 Multihoming Spine Spine Spine Spine
• Extended for VXLAN
• Host-side Overlay
• Dual-Connect Hosts VPC VPC
• Using Port-Channels Leaf VTEP VTEP Leaf Leaf VTEP VTEP
• Fabric-side
• Individual VTEPs
• Using a common Anycast VTEP
•
Seen as one VTEP from remote

Nodes
VPC Gateway Redundancy – A VXLAN perspective
Both sharing an
Overlay Anycast VTEP
VPC
VTEP VTEP
Individual Node Individual Node

with unique with unique
Identity Identity
Baremetal
Overlay
VPC
interface loopback0 VTEP VTEP interface loopback0

description RID description RID

description VTEP description VTEP
ip address 10.200.200.123/32 secondary ip address 10.200.200.123/32 secondary
Anycast VTEP Anycast VTEP

IP Address IP Address
Baremetal
Host Advertisements with VPC
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123
Overlay
VPC VPC
Leaf VTEP VTEP Leaf Leaf VTEP VTEP

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Host Advertisements with VPC
• Independent Devices
2
in0000.3001.1101
the EVPN / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123
Control-Plane Spine Spine Spine Spine
• Individual Router and

2
Peering
0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123
• Unique Route Distinguisher (RD) Overlay

• Independent Underlay Routing
VPC VPC
Devices Leaf VTEP VTEP Leaf Leaf VTEP VTEP
• Common VXLAN Device

• Next-Hop is Anycast VTEP
• Underlay ECMP Load Share to
Anycast VTEP Baremetal Baremetal Baremetal

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
ECMP to the Anycast VTEP – Underlay
Spine
VTEP
VTEP
VPC
Baremetal
AS#65500 101010110101010
10101010
Baremetal
VTEP
Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Spine
Bridging to a VPC Domain – VXLAN
Payload
10.200.200.104 10.200.200.123 3001 0000.3001.1102 0000.3001.1101 192.168.10.102 192.168.10.101
Spine
VTEP
VTEP
VPC
Baremetal
AS#65500 Baremetal
VTEP
Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Spine
Routing to a VPC Domain – VXLAN
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.10.101
Spine
Local Station
or
VTEP Virtual MAC
VTEP
VPC
Baremetal
AS#65500 Baremetal
VTEP
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Spine
• VPC provides Layer-2 Gateway
Redundancy
• From the VXLAN perspective,
the next-hop is always the
Anycast VTEP (VIP)
• Optimal for direct attached Hosts
• 1:1 Multicast-to-VNI mapping
• VPC operates at Layer-2

• MAC is Synchronized
• Local IP (ARP) is Synchronized
• Routing Tables are not
Synchronized
Subnet Route Advertisement with VPC
• Subnet Route Advertisement 5 192.168.11.0 /24

Spine
5000, 65500:5000
Spine Spine
10.200.200.123
Spine
• Route Type 5 5 192.168.22.0 /24 5000, 65500:5000 10.200.200.123

• Next-Hop is Anycast VTEP
Overlay
• Ensure Sync of Subnet VPC VPC
• Dual-Connect Networks (Point-2- Leaf VTEP VTEP Leaf Leaf VTEP VTEP
Point not Layer-3 over VPC)

• Synchronize Routing Table
• Advertise Route Type 5 with
individual VTEP IP (PIP) Baremetal
Host B
Subnet Y MAC: 0000.3001.1102
192.168.22.0/24
IP: 192.168.10.102
Subnet X
192.168.11.0/24
Subnet Route Advertisement with VPC
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101
Spine
VTEP
Subnet X VTEP
VPC
192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal
VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Spine
VPC – Dual-Attach Networks
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101
Spine
VTEP
Subnet X VTEP
VPC
192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal
VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Layer-3 Point-2-Point Spine
(not Layer-3 over

VPC!)
VPC – Synchronizing the Routing
Payload
10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101
Spine
VTEP
Subnet X VTEP
VPC
192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal
VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Dedicated Routing Spine
Session (per-VRF)
VPC – Advertise Subnet Individually (Advertise-PIP)
Payload
10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101
Spine
VTEP
Subnet X VTEP
VPC
192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal
VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Type IP / Length L3VNI / RT Next-Hop Seq. Spine
5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102
VPC – Advertise Subnet Individually (Advertise-PIP)
10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101 Payload

Spine
10.200.200.107 10.200.200.103 5000 0200.0ade.de07 0200.0ade.de03 192.168.20.101 192.168.11.101
VTEP
Subnet X VTEP
VPC
192.168.11.0/24 AS#65500 101010110101010

10101010
Baremetal
VTEP
Host C
MAC: 0000.3002.2101
IP: 192.168.20.101
Type IP / Length L3VNI / RT Next-Hop Seq. Spine
5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102
5 192.168.11.0/24 5000, 65500:5000 10.200.200.103
Agenda

• Overlay
• Underlay

• Multi-Tenancy
• VPC
A Deployment Story
Scalable Data Center Fabric
• VXLAN based Data Center Fabric
• BGP EVPN Control-Protocol (Overlay)
• OSPF for Underlay Routing (Unicast)
• PIM ASM with Anycast-RP for BUM Replication (Underlay)
• Distributed IP Anycast Gateway
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story RP Agg: 10.254.254.0/24
Underlay
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story – Underlay Routing RP Agg: 10.254.254.0/24
ip router ospf UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
router ospf UNDERLAY router ospf UNDERLAY

router-id 10.10.10.101 Spine Spine Spine Spine router-id 10.10.10.201
interface Ethernet1/1 interface Ethernet1/1

mtu 9192 mtu 9192
ip address 10.1.1.1/30
ip ospf network point-to-point
Underlay ip address 10.1.1.2/30
… Leaf Leaf Leaf Leaf Leaf Leaf Leaf
interface Ethernet1/2
mtu 9192
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
mtu 9192
ip address 10.1.1.10/30
ip pim sparse-mode
…
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
interface loopback0
router ospf UNDERLAY
ip address 10.10.10.102/32 router ospf UNDERLAY
router-id 10.10.10.101
ip router ospf UNDERLAY area 0.0.0.0 Spine Spine Spine Spine router-id 10.10.10.201
router ospf UNDERLAY interface Ethernet1/1
mturouter-id
9192 10.10.10.102 mtu 9192
ipinterface
ospf network point-to-point
Ethernet1/1
ip mtu
router
9192ospf UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
ip ip
pimaddress
sparse-mode
10.1.1.5/30 ip pim sparse-mode
… ip ospf network point-to-pointLeaf Leaf Leaf Leaf Leaf Leaf Leaf
ip router ospf UNDERLAY area 0.0.0.0 interface Ethernet1/2
ip pim sparse-mode mtu 9192
… ip address 10.1.1.6/30
ip pim sparse-mode
mtu 9192
ip address 10.1.1.10/30
ip pim sparse-mode
…
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
interface loopback0
router ospf UNDERLAY
ip address 10.10.10.102/32 router ospf UNDERLAY
router-id
ip router 10.10.10.101
ospf UNDERLAY area 0.0.0.0 Spine Spine Spine Spine router-id 10.10.10.201
interface loopback0
interface
router Ethernet1/1
ospf
ip addressUNDERLAY
10.10.10.103/32 interface Ethernet1/1
mturouter-id
9192
ip router10.10.10.102
ospf UNDERLAY area 0.0.0.0 mtu 9192
ipinterface
ospf
router network
ospf point-to-point
Ethernet1/1
UNDERLAY
ip mtu
router
9192ospf 10.10.10.103
router-id UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
ip ip
pimaddress
sparse-mode
10.1.1.5/30 ip pim sparse-mode
… ipinterface
ospf network point-to-pointLeaf
Ethernet1/1 Leaf Leaf Leaf Leaf Leaf Leaf
ip mtu
router
9192ospf UNDERLAY area 0.0.0.0 interface Ethernet1/2
ip ip
pimaddress
sparse-mode
10.1.1.9/30 mtu 9192
… ip ospf network point-to-point ip address 10.1.1.6/30
ip router ospf UNDERLAY area 0.0.0.0 ip ospf network point-to-point
ip pim sparse-mode ip router ospf UNDERLAY area 0.0.0.0
… ip pim sparse-mode
mtu 9192
ip address 10.1.1.10/30
ip pim sparse-mode
…
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203 Underlay ip pim anycast-rp 10.254.254.1 10.254.254.202
ip pim anycast-rp 10.254.254.1 10.254.254.203
ip pim rp-address 10.254.254.1 ip pim rp-address 10.254.254.1

Hypervisor Baremetal
interface
Hypervisor
loopback0
Hypervisor Baremetal Hypervisor Baremetal Baremetal
ip address 10.10.10.101/32
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203

interface loopback0
ip address 10.10.10.102/32
ip router ospf Baremetal
UNDERLAY Hypervisor
area 0.0.0.0
interface
Hypervisor
loopback0
Hypervisor Baremetal Baremetal
ip address 10.10.10.101/32
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203

interface loopback0
ip address 10.10.10.103/32
interface loopback0
ip address 10.10.10.102/32
ip router ospf Baremetal
UNDERLAY Hypervisor
area 0.0.0.0
interface
Hypervisor
loopback0
ip address 10.10.10.101/32
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story – the VTEP RP Agg: 10.254.254.0/24
Underlay
interface loopback1
ip address
Hypervisor
10.200.200.101/32
interface nve1
source-interface loopback1
host-reachability protocol bgp
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
Underlay
interface loopback1
ip address 10.200.200.102/32
interface loopback1
ip address
Hypervisor
10.200.200.101/32
ip routerinterface
ospf UNDERLAY nve1 area 0.0.0.0
interface nve1 host-reachability protocol bgp
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
Underlay
interface loopback1
ip address 10.200.200.103/32
interface loopback1
ip address 10.200.200.102/32
interface loopback1
ip routerinterface
ip address 10.200.200.101/32
Hypervisor Baremetal Hypervisor Hypervisor
source-interface
Baremetal
loopback1
ip routerinterface
interface nve1 host-reachability protocol bgp
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story – Overlay Control-Plane RP Agg: 10.254.254.0/24
router bgp 65500 router bgp 65500
router-id 10.10.10.202 router-id 10.10.10.203
neighbor 10.10.10.0/24 remote-as 65500 neighbor 10.10.10.0/24 remote-as 65500
update-source loopback0 update-source loopback0
address-family l2vpn evpn address-family l2vpn evpn
send-community both send-community both
route-reflector-client Spine Spine Spine Spine
route-reflector-client
Underlay
router bgp 65500

router-id 10.10.10.101
neighbor 10.10.10.202 remote-as 65500
update-source loopback0
address-family l2vpn evpn
Hypervisor Baremetal send-community
Hypervisor both

address-family l2vpn evpn
send-community both
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
Underlay
router bgp 65500

router bgp router-id
65500 10.10.10.102
router-id neighbor
10.10.10.101 10.10.10.202 remote-as 65500
update-source
neighbor 10.10.10.202 loopback0
remote-as 65500
update-source loopback0 l2vpn evpn
address-family
address-family send-community
l2vpn evpn both
Hypervisor neighbor
Hypervisor 10.10.10.203
both Baremetal remote-as
Hypervisor 65500
Baremetal Baremetal
update-source
neighbor 10.10.10.203 remote-as 65500 loopback0
update-source address-family
loopback0 l2vpn evpn
address-family send-community
l2vpn evpn both
send-community both
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
Underlay
router bgp 65500
router bgp router-id
65500 10.10.10.103
router-id neighbor
10.10.10.102 10.10.10.202 remote-as 65500
router bgp 65500
router-id neighbor
10.10.10.101 10.10.10.202 remote-as 65500
address-family send-community both
send-communityneighbor 10.10.10.203 remote-as 65500
address-family l2vpn evpn both
neighbor update-source
10.10.10.203 loopback0
remote-as 65500
Hypervisor both
address-family send-community both
address-family l2vpn evpn both
send-community
send-community both
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story – Layer-2 Service RP Agg: 10.254.254.0/24
vlan 100
vn-segment 30001
name Blue
vlan 200
vn-segment 30002
name Green
evpn
vni 30001
Overlay rd auto
route-target both auto
vni 30002
rd auto
route-target both auto
interface nve1
member vni 30001
mcast-group 239.239.239.1
member vni 30002
mcast-group 239.239.239.2
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story – Layer-3 Service RP Agg: 10.254.254.0/24
vlan 2001 vrf context VRF-A
vn-segment 50001 vni 50001
rd auto
interface Vlan2001 address-family ipv4 unicast
mtu 9192 route-target both auto
vrf member VRF-A route-target both auto evpn
ip forward Spine Spine Spine Spine
address-family ipv6 unicast
no ip redirects route-target both auto
route-target both auto evpn
Overlay interface nve1
member vni 50001 associate-vrf
router bgp 65500

vrf VRF-A
advertise l2vpn evpn
redistribute direct route-map TAG
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story – First-Hop Gateway RP Agg: 10.254.254.0/24
interface Vlan100 router bgp 65500
mtu 9192 vrf VRF-A
vrf member VRF-A address-family ipv4 unicast
ip address 192.168.1.1/24 tag 21921 advertise l2vpn evpn
fabric forwarding mode anycast-gateway redistribute direct route-map TAG
interface Vlan200 Spine Spine Spine Spine

route-map TAG permit 10
mtu 9192 match tag 21921
vrf member VRF-A
ip address 10.10.10.1/24 tag 21921
fabric forwarding mode anycast-gateway Overlay
p2p Agg: 10.1.1.0/24
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
A Deployment Story RP Agg: 10.254.254.0/24
interface Ethernet 2/1.10
vrf member VRF-A interface Ethernet 1/15.21
ip address 172.16.0.1/30 vrf member VRF-A
encapsulation dot1q 5 ip address 172.16.0.2/30
encapsulation dot1q 5
vrf member VRF-B Spine Spine Spine Spine
ip address 172.16.0.1/30 vrf member VRF-B
encapsulation dot1q 6 ip address 172.16.0.2/30
router bgp 65500 Overlay encapsulation dot1q 6
vrf VRF-A router bgp 65599
address-family ipv4 unicast vrf VRF-A
advertise l2vpn evpn address-family ipv4 unicast
aggregate-address 10.10.10.0/24 summary-only neighbor 172.16.0.1 remote-as 65500
aggregate-address 192.168.1.0/24 summary-only update-source Ethernet1/15.21
neighbor 172.16.0.1 remote-as 65599 address-family ipv4 unicast
update-source Ethernet2/1.10 …
…
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal

Subnet B
192.168.20.0/24
Summary
Summary
• Multi-Tier Topologies based on Leaf and Spines (aka Clos)
• New paradigm with Hierarchical Overlays
• Overlays (VXLAN) for Network Virtualization
• Different flavors of Overlay Solution (Flood&Learn and BGP EVPN)
• Layer-3 in the Underlay – Defines the Topology
• Layer-2 and Layer-3 in the Overlay – Defines the Services
• End-Points State exists in the Overlay
• BGP EVPN for integrated Layer-2 and Layer-3 Services

• Control-Plane driven
• Optimal Routing and Bridging
• Avoid hair pinging and reduced failure domains
BRKDCN-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
If you haven’t
had enough
VXLAN BGP
EVPN
Links & Resources
• VXLAN Multi-Site Intro
• https://blogs.cisco.com/datacenter/vxlan-innovations-vxlan-evpn-multi-site-part-2-of-2
• VXLAN Multi-Site @ Cisco Live online
• https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035#/
• ”eBGP” for EVPN
• https://learningnetwork.cisco.com/blogs/community_cafe/2017/11/02/vxlan-ebgp-evpn-
the-incarnation-of-a-hybrid-guest-post
• Configuration Example
• https://communities.cisco.com/community/technology/datacenter/data-center-
networking/blog/2015/05/19/vxlanevpn-configuration-example
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKDCN-2949
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
www.ciscolive.com/global/on-demand-library/.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions
BRKDCT-2949 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 154
Thank you

BRKDCT 2949

Uploaded by

Copyright:

Available Formats

You might also like

BRKDCT 2949

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 2949

Uploaded by

Copyright:

Available Formats

BRKDCT-2949

Building Data Center

Lukas Krattiger, Principal Engineer

• Introduction to Data Center Fabric

• VXLAN with BGP EVPN

Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2 Layer-2

• Introduction to Data Center Fabric

• VXLAN with BGP EVPN

Spine Spine Spine Spine

• Wide ECMP: Unicast or Multicast

More Spine – More Bandwidth – More Resiliency

• Smallest Operational Entity

Spine Spine Spine Spine Spine Spine Spine Spine

• Not Limited to Port Density

• Beyond a single Server Room

• Retains Intra-Pod Topology with

Spine Spine Spine Spine

• Any Subnet, Anywhere, Rapidly

• Reduced Failure Domain

• Extensible Scale and Resiliency

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

• Physical and Virtual

• Introduction to Data Center Fabric

• VXLAN with BGP EVPN

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

• A Single Overlay Domain

Overlay • External Connectivity

• @ Leaf = Border Leaf

Nicely Structured and Tiered Topologies

• Scale-Out Model to Build a Large

Data Center Interconnect (DCI)

VXLAN Multi-Pod VXLAN Multi-Fabric

Overlay Overlay Overlay Overlay

Bar Bar Bar Bar

 Single Fabric with End-to-End  Multiple Fabrics – Normalized

VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site

EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Overlay Overlay Overlay Overlay Overlay Overlay

Bar Bar Bar Bar

• Multi-Site and/or External

@ Spine = Border Spine

Your Happy Place! 

• Introduction to Data Center Fabric

• VXLAN with BGP EVPN

• Network Switches support MTU

• Routed Ports and Interfaces

and Spine(no switchport)

• Alternative, use IP Unnumbered

• Use Loopback as Source-

• Unicast Routing – Routing

p2p Agg: 10.1.1.0/24

• Example from depicted Topology

• 11 Router ID (RID Loopback)

• 11 Router ID (RID) = 11 IP Addresses

• Total: 74 IP Addresses Required

• Example from depicted Topology

• 11 Router ID (RID Loopback)