VXLAN Fundamentals, Architecture &

Roadmap

1
Table of Contents
1. Data Center IP Fabric ‘Building a strong Foundation’
2. What is ‘Network Virtualization’?
3. VXLAN Overview
4. VXLAN Packet details
5. VXLAN Terminology
6. VXLAN Host Discovery
7. VXLAN BUM Traffic Handling
8. VXLAN Layer 2 & Layer 3 Terminologies
9. VXLAN Arista Architecture & Vision
10. VXLAN Roadmap
11. VXLAN Visbility

2
 Data Center – ‘IP Fabric’
Building A Strong Foundation

3
Challenges with current network architecture
Legacy Data Center Model
Oversubscription
 Ports on devices are oversubscribed ~ 8:1
 Higher Oversubscription as traffic traverses
north ~ 20:1

Scalability
 Scales up and not scales out

North to South
 Dependent on specific hardware (mix &
match)
 Not scalable to 40GbE / 100GbE

Cost
 As multiple layers, it can get $$$

Mobility
 What happens if my “IP” changes?
 What happens if traffic pattern changes?

Layer 2 Layer 2 Layer 2 Layer 2

Domain Domain Domain Domain

Multiple points of management,

Latency
rampant oversubscription, wasteful  High latency
cost model  Low predictability
 Data Center ‘IP Fabric’
 Support for East/West 80:20 traffic
pattern
 Scale up to 64-way ECMP Spine designs
 All uplinks from ToR are Active/Active
 Support 100’000s of host ports
 Non-blocking / Non-oversubscribed
architecture
 Deploy L3 routing protocols between leaf
& spine i.e. BGP, OSPF, or ISIS
 Everything is only 3 hops away!
 Provide network mobility via ‘Overlay
Network’
 Arista – Spine/Leaf “IP Fabric” Architecture

Spine Tier  

IP Fabric
Leaf Tier VTEP3 VTEP4

VTEP1 VTEP2

HYPERVISOR 1 HYPERVISOR 2

A1 B1 A2 B2
Bare Metal Bare Metal
Storage Servers

 Network core is an IP fabric laid out in a Leaf-Spine architecture running

ECMP between the two tiers
- Leaf switches - Arista 7150-x or 7050Q-x models are deployed at the TOR connecting
virtualized servers, bare-metal servers, storage arrays and other devices
- Spine switches – Arista 7500’s are deployed at the core
- Routing Protocol – Either EGP (BGP) or IGP (OSPF / ISIS) is run in the IP fabric
6
What is Network Virtualization?

7
What is Network Virtualization?
Network Virtualization is not the same as Server Virtualization!

8
 Overlays v Underlays
Network virtualization: ability to separate, abstract and decouple the physical
topology from a ‘logical’ or ‘virtual’ topology by using encapsulated tunneling.

Overlay
Network

Physical
Infrastructure i.e.
Underlay Network

This logical network topology is often referred to as an ‘Overlay Network’.

VXLAN disassociates workloads from physical networks, allowing for

possible transition to cloud based providers
9
Types of ‘Overlay’ Technologies
Any Overlay technology uses Location & Identity separation

Location

Identity

Fabric Path VXLAN OTV LISP

Underlay Protocol IS-IS BGP, OSPF, IS-IS BGP, OSPF, IS-IS BGP, OSPF, IS-IS

Location Switch-ID IP address IP address IP address

Identity Client MAC Client MAC Client MAC Client IP / Mac

Identity Learning Flooding Flooding / IS-IS Mapping DB

Dynamic learning
Vendor Proprietary Yes Non Yes Non

Intra & / or Inter DC Intra Both Both Inter

10
VXLAN Overview

11
 Virtual Extensible Local Area Network (VXLAN)

 Ethernet in IP overlay network  Tunnel between ESX hosts

• Entire L2 frame encapsulated in • VMs do NOT see VXLAN ID
UDP
• 50 bytes of overhead  IP multicast used for L2
broadcast/multicast, unknown
 Include 24 bit VXLAN Identifier unicast
• 16 M logical networks  Technology submitted to IETF
 VXLAN can cross Layer 3 for standardization
• With Arista, Vmware, Red Hat, Citrix,
Cisco, and Others

Outer Outer Inner Inner Optional Original

Outer Outer Outer Outer VXLAN ID
MAC MAC MAC MAC Inner Ethernet CRC
802.1Q IP DA IP SA UDP (24 bits)
DA SA DA SA 802.1Q Payload

VXLAN Encapsulation Original Ethernet Frame

12
Virtual eXtensible LAN: How does it
work?
VM-2
VM-1 Layer 2 Domain between the VM
10.10.10.1/24 10.10.10.2/24
vWire- VNI 10

VTEP VTEP

Subnet-A Subnet-B

SW VTEP HW VTEP
Encap/Decap MAC&IP are UDP Encapsulated Encap/Decap
VXLAN VTEP VXLAN Frames

Encapsulation at VTEP node is transparent to IP ECMP fabric

13
 VXLAN Benefits
 Feature Benefits
- Eliminates current networking challenges in the way of on-demand, virtual
environment:
- VLAN Sprawl
- Single fault domains
- Scalability beyond 4096 segments
- Proprietary fabric solutions
- IP mobility
- Physical cluster size and locality
- Enables multi-tenancy at scale
- Decouples logical networks from physical infrastructure so that applications can
be deployed without worrying about physical rack location, IP address or VLAN
- Based on open and well known standards

14
VXLAN Use Cases
 Physical to Virtual internetworking
 Multi-hypervisor connectivity and integration
 Multi-tenant Cloud environments
 HA clusters across failure domains
 Dynamic growth
 Dynamic resource management

15
VXLAN Packet Details

16
VXLAN Packet
VXLAN is a MAC-in-IP encapsulation

17
 VXLAN Header
VXLAN Header is a 8 Byte field comprising of:
(a)Flags (8 Bits)
(b)VxLAN Network Identifier (VNI) (24 Bits)
(c)Reserved (24 & 8 Bits) – Always set to zero.

Flags (8 Bits) – I flag is set to 1 for a valid VxLAN

Network ID (VNI). The remaining 7 bits (designated "R")
are reserved fields and set to zero.

VxLAN Network Identifier (VNI) (24 Bits) – Used for

identification of the individual VxLAN overlay network on
which the communicating VMs are situated. VMs in
different VxLAN overlay networks cannot communicate.

Reserved (24 & 8 Bits) – Always set to zero.

18
VXLAN Terminology

19
 VXLAN Terminology – Physical Topology

Spine Tier   Hardware VTEP

IP Fabric
Leaf Tier VTEP3 VTEP4

VTEP1 VTEP2

Software VTEP HYPERVISOR 1 HYPERVISOR 2

A1 B1 A2 B2
Bare Metal Bare Metal
Storage Servers

VTI VTI
VXLAN Segments VXLAN Gateway
VXLAN
10001

VXLAN
10002

20
 VXLAN Terminology – Logical Topology
External
Host

Data
Center
Network

VARP
VARP VARP
VXLAN Segment Default
DefaultGateway:
Gateway: Default Gateway: VXLAN Segment
10.100.1.1
10.100.1.1 10.100.2.1
VNI
VTEP 1 VTEP 3 VTEP 1 VTEP 4

.1 .1 .1 .1
10.100.1.0/24 VXLAN 10001 10.100.2.0/24
VXLAN 10002
.2 .10 .11 .2 .3 .10
B1 B2 A1 A2

Bare Metal
Storage
Bare Metal
Servers

21
VXLAN Terminology Explained
 VTEP: VXLAN Tunnel End Point
- VXLAN encapsulation and decapsulation happens at the VTEP

 VXLAN Gateway
- A device which bridges traffic from VXLAN and non-VXLAN environments.
- VXLAN gateways allow for physical and non virtualized devices to communicate with VXLAN
networks
- A VXLAN gateway can be either a hardware or software device
 VNI: Virtual Network Identifier
- a 24-bit number is also called the VXLAN segment ID. The system uses the VNI, along
with the VLAN ID, to identify the appropriate tunnel.
 VXLAN Header – is an 8-byte header that contains the 24-bit VNI value. It lives in between the UDP header
and the inner MAC frame being carried over the VTI.

 VTI: VTEP Tunnel Interface - a switchport linked to a UDP socket that can be shared between many
VLANs. Packets bridged through a vlan into the VTI are sent out the UDP socket with a VXLAN header including a
VNI. The socket is bound to a fixed local port, but is not connected to any particular destination port or IP address;
logically, we use sendto() (not send()) to transmit VXLAN-encapsulated frames on the socket. Packets arriving on the VTI
(via the UDP socket, based on their UDP destination port) are demultiplexed into a VLAN for bridging. A 24-bit VNI
within the packet determines which VLAN the packet is mapped to for bridging.

 VXLAN Segment - is a Layer 2 overlay network over which VMs communicate. Only VMs within the same
VXLAN segment can communicate with each other.

22
VXLAN Visibility

23
VXLAN Visibility - Arista’s vmTracer

 Full physical to virtual visibility

 Network audit to ensure

reachability

 Automated provisioning

 Workflow without finger pointing

 Other awesome capabilities

24
 Monitoring VXLANs with vmTracer
Virtualization

 Rapidly correlate vlan to VNI

switch5#:show vmtracer vxlan interface Ethernet48
Ethernet48: esx1.aristanetworks.com/ndsTest/dvuplink1
VM Name VLAN vWire Network Multicast
--------------------------------------------------------------------------------------------
Exchange 5 Corp 172.20.20.0 239.20.20.0
Apache 6 web 182.10.0.0 220.10.10.0
vmTracer
MySQL 7 ERP 172.20.30.0 239.20.30.0

 view VNIs across the data center from the CLI

switch9#:show vmtracer vxlan all
7150s R1: Ethernet 48:esx1/vwTest/dvUplink 1
vWire:Corp -- VLAN:5
vWire:ERP -- VLAN:7 VTEP VTEP VTEP VTEP
7150s R2: Ethernet 40:esx2/vwTest/dvUplink 1
vWire:Corp -- VLAN:5
vWire:web -- VLAN:6 Hypervisor
VMware NSX Physical

25
 Automate Learning of VNI State
NSX Controller

New VNI - CalBears

Multicast Group - 224.0.14.13
VNI ID - 650782

Interface Ethernet 24
VXLAN VTEP VNI CalBears

Interface Loopback0
VXLAN VTEP Gateway VNI Calbears
IP Address 204.181.40.1/24

<--Network

VM- Oski
VNI - CalBears
26
Where is my VM now?
spine0: show vmtracer vxlan
VNI-Name VNI #VTEPs Learning Mcast Group Status
Subnet
Auburn 5096 4 Flood 224.0.1.95 Up
204.181.40.0/24
foo 15893425 5 Flood 224.0.4.84 Up 128.218.56.0/24
bar 65456 45 Flood 224.5.1.92 Down
192.168.10.0/20
spine0: show vmtracer vxlan vni Auburn
spine0
VNI Name: Auburn
VNI Segment ID: leaf15096 leaf2
VTEP Type Status Inside Outside Learning Mcast Grp PIM-RP
Switch Port Model
ESX1 VMware Up 3 VNICs 204.181.21.5 Flood 224.0.1.95 204.181.1.16
ar16 eth15 7050S
ar24 Arista Up/GW 204.181.40.1 204.181.1.16 Flood 224.0.1.95 204.181.1.16
ar24 loop0 7150S
ar22 Arista Up/Up 1 MAC/IPs 204.181.3.67 Flood 224.0.1.95 204.181.1.16
ar22 eth2 7150S
ESX4 VMware Up 4 VNICs 204.181.1.5 Flood 224.0.1.95 204.181.1.16
ar2 eth23 7050T
esx10 esx11
VNI ‘Test’: 224.0.0.12

Aubie WarEagle vshield vm-tiger 27

Where is my VM now?

spine0: show vmtracer interface vxlan Auburn

VTEP: ESX1 Role: vSwitch Switch/Port: ar16.foo.com/eth15

Name VNIC Status State IP Address
Aubie Network Interface 1 Up/Up vMotion 204.181.40.2
WarEagle Network Interface 2 Up/Up VM-FT-A 204.181.40.3
BooBama Network Interface 1 Up/Down -- 204.181.40.5

VTEP: ar24 Role: Router

spine0
Switch/Port: ar24.foo.com/loopback0
NAT/PAT leaf1 Status #ARPs IP Address leaf2
No Up/Up 45 204.181.40.1

VTEP: ar22 Role: Port-VTEP Switch/Port: ar22.foo.com/eth2

FQDN IP MAC VLAN Status
isilon16.foo.com 204.181.40.190 00-00-45-ab-12-fe 5 Up/Up

128.218.10.x 128.218.11.x

esx1 esx11
VNI ‘Test’: 224.0.0.12

Aubie WarEagle vshield vm-tiger 28

THANK YOU

29