LEARN WORK IT

INFORMATION TECHNOLO GY (NE T WORK )

C I S C O ACI BLO GS VMWARE N SX BLO G S CISCO ROUT ING B LO G

C I S CO SW ITCHIN G BLO G IT INS TITU TES CONTACT US

TERMS & CONDIT ION

6. ACI VXLAN
 APRIL 8, 2021

VXLAN in ACI
VXLAN is an industry-standard protocol that extends
Layer 2 segments over Layer 3 infrastructure to build
Layer 2 overlay logical networks. The ACI infrastructure
Layer 2 domains reside in the overlay, with isolated
broadcast and failure bridge domains. This approach
allows the data center network to grow without the risk
of creating too large a failure domain.
All traf�c in the ACI fabric is normalized as VXLAN
packets. At the ingress,ACI encapsulates external VLAN,
VXLAN, and NVGRE packets in a VXLAN packet. The
following �gure shows ACI encapsulation
normalization.

Every packet in the fabric carries ACI policy attributes,

ACI can consistently enforce policy in a fully distributed
manner. ACI decouples application policy EPG identity
from forwarding. The following illustration shows how
the ACI VXLAN header identi�es application policy
within the fabric.

The ACI VXLAN packet contains both Layer 2 MAC

address and Layer 3 IP address source and destination
�elds, which enables ef�cient and scalable forwarding
within the fabric. The ACI VXLAN packet header source
group �eld identi�es the application policy endpoint
group (EPG) to which the packet belongs. The VXLAN
Instance ID (VNID) enables forwarding of the packet
through tenant virtual routing and forwarding (VRF)
domains within the fabric. The 24-bit VNID �eld in the
VXLAN header provides an expanded address space for
up to 16 million unique Layer 2 segments in the same
network. This expanded address space gives IT
departments and cloud providers greater �exibility as
they build large multitenant data centers.
VXLAN enables ACI to deploy Layer 2 virtual networks
at scale across the fabric underlay Layer 3
infrastructure.

VXLAN Forwarding in ACI

ACI performs L2 and L3 traf�c forwarding on VXLAN

Overlay. In ACI leaf nodes are called as PTEPs ( Physical
Tunnel End Points). But in general leaf switches are
called as VTEPs (VXLAN Tunnel Endpoints). In ACI
Layer 2 switched traf�c carries a VXLAN Network
Identi�er (VNID) to identify bridge domains, and Layer 3
(routed) traf�c carries a VRF ID in VNID. The
encapsulation/decapsulation of the VXLAN header is
done on VTEP.
VXLAN also allows mapping of the location to the
identity of endpoints. In Cisco ACI, the endpoint’s IP
address is the identi�er, and a VTEP address designates
the location (leaf) where endpoints are connected. Cisco
ACI uses a dedicated VRF and interfaces of the uplinks
as the infrastructure to carry VXLAN traf�c. The
transport infrastructure for VXLAN traf�c is known as
Overlay-1, which exists as part of tenant Infra.

The Overlay-1 VRF in ACI contains /32 routes to each

VTEP, vPC virtual IP address, APIC as well as spine
proxy IP address.

P T E P I P a d d r e s s :- This is the IP address provided by

APIC from Infrastructure Subnets as a loopback
interface, which was con�gured on APIC initial
con�guration phase.This address is used for
communication with APIC , other Leafs , MP-BGP
peering, traceroute or ping.
P r o x y T E P I P a d d r e s s :- This is an anycast IP
address that is present across all spines and is used for
forwarding lookups into the mapping database.
F T E P I P a d d r e s s :- This address is used when VMM
domain ( ESXI environment ) is present. A fabric
loopback TEP (FTEP) is used to encapsulate traf�c in
VXLAN to a vSwitch VTEP . It is a unique FTEP address
that is identical on all leaf nodes to allow mobility of
downstream VTEP devices.
v P C l o o p b a c k V T E P a d d r e s s :- This IP address is
used when the two leaf nodes forward traf�c that
enters through a vPC port. Traf�c is forwarded by the
leaf using the VXLAN encapsulation. This address is
shared with the vPC peer.

VXLAN Headers for ACI Fabric

In the ACI fabric, some extensions have been added to

the VXLAN header to support the following features in
ACI :
Security zones segmentation ( Tenant )
Management of �ltering rules and policies ( Contracts /
Filters )
Enhanced load-balancing techniques
The VXLAN header used in the Cisco ACI fabric is
shown below :
When any packet uses VXLAN in ACI then the
Minimum MTU size that the fabric ports need to
support is the original MTU (1500) + 50 bytes.
Original MTU ( 1500) + 14 Bytes ( Frame ) + 20 Bytes ( IP
Header ) + 8 Bytes ( UDP) + 8 bytes ( iVXLAN) = 1550
bytes
The Cisco ACI fabric uplinks are con�gured for 9150
bytes, which is large enough to accommodate the traf�c
of servers sending jumbo frames. The MTU of the fabric
access ports is 9000 bytes, to accommodate servers
sending jumbo frames. Cisco uses some more bits and
spaces in the VXLAN header to use it in its ACI
infrastructure. In the VXLAN header, Cisco Uses the
following more �eld: Source Group: To determine the
Source EPG P bit called Policy bit, When its value is set
to 0, Policy is not instantiated on leaf and if its value is 1
then the policy is instantiated.

Layer 3 VNIDs Facilitate

Transporting Inter-subnet Tenant
Traf�c

The ACI fabric provides tenant default gateway

functionality that routes between the ACI fabric VXLAN
networks. For each tenant, the fabric provides a virtual
default gateway that spans all of the leaf switches
assigned to the tenant. It does this at the ingress
interface of the �rst leaf switch connected to the
endpoint. Each ingress interface supports the default
gateway interface. All of the ingress interfaces across
the fabric share the same router IP address and MAC
address for a given tenant subnet. The ACI fabric
decouples the tenant endpoint address, its identi�er,
from the location of the endpoint that is de�ned by its
locator or VXLAN tunnel endpoint (VTEP) address.
Forwarding within the fabric is between VTEPs. The
following �gure shows decoupled identity and location
in ACI.

VXLAN uses VTEP devices to map tenant end devices to

VXLAN segments and to perform VXLAN encapsulation
and de-encapsulation. Each VTEP function has two
interfaces:
• A switch interface on the local LAN segment to
support local endpoint communication through
bridging
• An IP interface to the transport IP network

Control Protocols

Following are the control-plane protocols

running inside the fabric:

Interme diate Switch–to–Interme diate Switch

( I S - I S ) p r o t o c o l runs on the interfaces between leaf
and spine to maintain infrastructure reachability.
Council of Oracles Protocol reference d as
( C O O P ) runs on the loopback address of PTEP to
synchronize and it ensures the consistency of the
endpoint database or Mapping table on spine
switches.COOP de�nes roles to spine and leaf. All spines
are called Oracle and all leafs are called Citizens. If
anything is learned by Citizens they will inform to
Oracles and if anything is learned by Oracles, that will
be informed to all Oracles.
M P - B G P also runs on the PTEP loopback and it
advertises all external WAN routes throughout the
fabric.
V X L A N t u n n e l s are created between PTEPs of other
leaf and spine proxy TEPs.
Each leaf maintains the VXLAN tunnels database with
all other leaf nodes on Overlay-1.