ACI Troubleshooting Multipod

ACI Troubleshooting:
Multipod
Joseph Young, ACI Technical Leader
BRKACI-2934
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• Introduction
• Multipod Overview
• Troubleshooting the Multipod Setup Process
• Troubleshooting Unicast Flows
• Troubleshooting Multi-destination Flows
• Troubleshooting External Routed Communication
• Quality of Service
• Conclusion
BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Acronyms/Definitions
Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure MDT Multicast Distribution Tree
ACL Access Control List MST Multiple Spanning Tree
APIC/IFC Application Policy Infrastructure OSPF Open Shortest Path First Protocol
Controller/ Insieme Fabric Controller
BD Bridge Domain pcTag Policy Control Tag
COOP Council of Oracle Protocol PIM Protocol Independent Multicast
ECMP Equal Cost Multipath PL Physical Local
EP Endpoint SVI Switch Virtual Interface
EPG Endpoint Group TC Topology Change

EVPN Ethernet VPN BGP Address-family VL Virtual Local
FTEP/VTEP Fabric/Virtual or VXLAN Tunnel Endpoint VNID Virtual Network Identifier
GIPo Outer Group IP Address VPNv4 BGP VPN Address-Family
ISIS Intermediate System to Intermediate VXLAN/iVXLAN Virtual Extensible LAN / Insieme VXLAN
System
LPM Longest Prefix Match XR VXLAN Remote
Multipod Overview
Feature Evolution
I said no marketing…why is this necessary?
• Effective Troubleshooting requires understanding…

• Why does the feature exist?
• What problems does it solve?
• How does it solve them?
• How do the components interact?
Understand the “why”
Feature Evolution – Classic ACI
Spine Spine
• VXLAN TEP reachability
Single ISIS
learned through ISIS
Fabric COOP
MPBGP • Endpoint Repo on Spines
handled by COOP
• MP-BGP to distribute
Leaf Leaf Leaf Leaf external routes through fabric
APIC APIC APIC
Feature Evolution
What if ACI must be extended to other locations?
• TEP reachability must be communicated

• Endpoints must be synced across locations
• Mechanism needed for BUM traffic
• APIC Cluster must be extended
Feature Evolution – Stretched Fabric
Spine Spine Spine Spine
Single
Fabric
Leaf Leaf Leaf Leaf Leaf Leaf
APIC APIC APIC
Feature Evolution – Stretched Fabric
• Transit leafs connect to all spines
• COOP, ISIS, and BGP extended across locations
Not scalable
Feature Evolution – Multipod
IPN
OSPF,PIM
• Single Fabric Extended
BGP VPNv4/EVPN • Each pod is local
instance of ISIS and
ISIS Pod2 ISIS COOP
Pod1
COOP COOP
• Inter-pod connectivity
MPBGP MPBGP
through IPN
• Inter-pod BUM uses
Leaf Leaf Leaf Leaf Leaf Leaf PIM-Bidir
• BGP between pods to
share endpoints and
external routes
APIC APIC APIC
IPN Requirements
❑OSPF
❑DHCP relay
❑Jumbo MTU (9150 Bytes)
❑Routed Subinterfaces
❑PIM Bidir with at least /15 Mask
❑QoS (optional)
Troubleshooting
Multipod Setup
Multipod Setup Overview
1. Configure Pod 1 (TEP pool, infra l3out)

2. Configure Remote Pod (TEP pool, infra l3out)
3. Register Remote Pod Spines (DHCP)
4. Discover Remote Pod Leafs (LLDP)
5. Remote Pod APIC’s join cluster
Multipod Setup Process
Setting up Pod 1 (Seed Pod)
➢ Configure
Addressing for
Pod 1 Spine > IPN
connection
L3 Interface used for OSPF peering

with IPN. Make sure MTU matches IPN!
➢ Configure OSPF
parameters for
Pod 1 Spine > IPN
connection
➢ Configure
Dataplane TEP
Anycast Address used for Pod x > Pod 1

Proxied traffic. More on this in Unicast Section
Not needed for

Multipod, leave blank
if not used
➢ Review POD1
configurations
After setting up Seed Pod (Pod 1)…
✓ Verify OSPF is up between Pod1 Spines and IPN

pod1-spine1# show ip ospf neighbors vrf overlay-1
OSPF Process ID default VRF overlay-1
Total number of neighbors: 1
Neighbor ID Pri State Up Time Address Interface
172.31.255.1 1 FULL/ - 00:00:14 172.21.0.0 Eth1/23.23
✓ Ensure IPN is pre-provisioned for Pod 2 Connections

Configure Jumbo MTU interface Ethernet1/21.4
mtu 9150
Use Vlan 4 Subinterface encapsulation dot1q 4
vrf member P1IPN
ip address 172.22.0.0/31
ip ospf network point-to-point
Ensure PIM is Enabled ip router ospf P1IPN area 0.0.0.0
ip pim sparse-mode
ip dhcp relay address 10.0.0.1
Make sure dhcp relay is configured
ip dhcp relay address 10.0.0.2
pointing to APIC Infra IP’s
no shutdown
Setting up Pod 2
➢ Assign TEP pool to

non-seed Pod
➢ Configure L3
parameters
L3 Interface used for OSPF

peering with Pod 2 IPN.
➢ Configure OSPF
parameters
➢ Configure
Dataplane TEP for
Pod 2
Anycast Address used for Pod x > Pod 2

Proxied traffic. More on this in Unicast Section
Not needed for multipod,

leave blank if not used
Where to find the less-known MPOD configurations?
Leave default, allows PODs to

import BGP paths from each other
Dataplane TEPs
from Setup
Spine > IPN subnets

leaked into ISIS
➢ POD 2 Spines
should now be
discoverable
IPN Remote Pod Discovery
DISCOVER
Pod1 Pod2 1. Remote Pod

Spines send DHCP
DISCOVER. IPN
Relays to APICs
Leaf Leaf Leaf
APIC APIC
Pod1 Pod2
2. IP Address from
Multipod l3out is
assigned.
Leaf Leaf Leaf
OFFER
APIC APIC
What’s in the
DHCP OFFER?
• IP address from L3out Pod2 Facing IPN
IP address (relay)
interface profile
assigned
• Gateway is next-hop
for default route
Offered IP (From l3out
• Bootstrap file interface profile)
communicates
location of l3out Directory on APIC from which Spine downloads full
Config l3out configuration *full directory is
/firmware/fwrepos/fwrepo/boot/bootstrap-202.xml
Default Gateway, used

for downloading config
Pod1 Pod2
3. Spine configures
static default route
for APIC reachability
with NH of IPN.
Leaf Leaf Leaf pod2-spine2# vsh -c "show ip route 0.0.0.0/0 vrf overlay-1"
IP Route Table for VRF "overlay-1"
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
0.0.0.0/0, ubest/mbest: 1/0

APIC APIC *via 172.22.0.0, Eth1/23.23, [250/0], 5d17h, static
switch# moquery -c topSystem
# top.System
address : 0.0.0.0
bootstrapState : downloading-bootstrap-config
role : spine
Pod1 Pod2
4. Spine downloads
bootstrap XML from
APIC which contains
l3out configuration
Leaf Leaf Leaf
pod1-apic1# grep bootstrap /var/log/dme/log/access.log
CONFIG 172.22.0.1 - - [18/Apr/2019:14:13:16 +0000] "GET

/fwrepo/boot/bootstrap-202.xml HTTP/1.1" 200 8561 "-" "-"
APIC APIC 200 OK Code. GET

was success!
Lo0
DISCOVER
Pod1 Pod2
5. Spine acts as self
relay for TEP DHCP
request
6. TEP address from
Leaf Leaf Leaf POD2 pool is
assigned
OFFER
APIC APIC
LLDP
Spine Spine Spine Spine DHCP
Pod1 Pod2
7. Pod2 Leafs
discovered through
normal process
(LLDP/DHCP)
APIC APIC
Pod1 Pod2
8. Pod2 APIC(s) join
cluster
*Non-seed pod APICs
still use Pod1 TEP Pool!
APIC APIC APIC
Common Multipod Discovery Problems
Issue #1: Pod2 Spines Don’t Receive L3out IP or Config
Possible Causes
1. DHCP Relays on IPN point to APIC OOB rather than infra
✓Configure Relays to point to infra (show controller on APICs)
2. IPN doesn’t have route to APICs
✓Check that OSPF is up between IPN and Pod1
3. Miscabling results in Spine receiving IP in different subnet than GW
✓Correct cabling or addressing then remove and rediscover Spine
4. Spines can’t resolve ARP for connected IPN interface
✓Ensure SW version supports multipod + spine hw (ex: for 9364C MPOD
supported in 3.1(1))
Issue #2: Pod2 Spines Don’t Receive TEP Addresses
Ensure leafs are connected to spine
-Spine TEP not assigned until leaf-facing interfaces “up”
Ensure Leaf–facing
interfaces are “up”
so Spine gets TEP
Issue #3: Remote Pod APIC Not Joining Cluster
Check your setup Parameters! Cluster configuration ...

Enter the fabric name [ACI Fabric1]: CL-Fab
Enter the fabric ID (1-128) [1]: 1
Enter the number of active controllers in the fabric (1-9) [3]:
Ensure Pod ID is Enter the number of active controllers in the fabric (1-9) [3]: 3
Correct Enter the POD ID (1-12) [1]: 2
Is this a standby controller? [NO]:
Is this an APIC-X? [NO]:
Remote Pod APIC must Enter the controller ID (1-3) [1]: 3
use Pod 1 TEP Pool Enter the controller name [apic3]: p2-apic3
Enter address pool for TEP addresses [10.0.0.0/16]:
Note: The infra VLAN ID should not be used elsewhere in your
environment
✓ Run “acidiag avread” to check setup config and should not overlap with any other reserved VLANs on other
platforms.
✓ If wrong wipe and reload the APIC Enter the VLAN ID for infra network (1-4094): 3967
“acidiag touch clean” Out-of-band management configuration ...
“acidiag touch setup” Enable IPv6 for Out of Band Mgmt Interface? [N]:
Enter the IPv4 address [192.168.10.1/24]: 10.122.143.14/26
“acidiag reboot” Enter the IPv4 address of the default gateway [None]: 10.122.143.1
Enter the interface speed/duplex mode [auto]:
Multipod Setup Verification Checklist
❑Verify BGP EVPN and VPNv4 is up between pods
❑Verify both unicast and multidestination interpod flows work
❑Verify jumbo MTU interpod flows work
❑Verify above flows work during various Spine > IPN and IPN > IPN
link failures
Troubleshooting
Unicast Flows
Multipod Unicast Overview
Key Differences Between Single Pod Unicast
• Spines share EP’s via BGP EVPN

• Tunnels to remote pod dynamically built
• IPN’s must support overlay traffic requirements
Troubleshooting Single Pod and Multipod is Similar!
Layer 2 Unicast
BD Settings - UUC Proxy, ARP Flooding Enabled, UC Routing Disabled
Pod1 Pod2
Verify first that the
flow is unicast
EP1 root@vm1:/home/joyo# arp -an

172.16.1.1/24 ? (172.16.1.2) at 8c:60:4f:02:88:fc [ether] on ens192
0050.56a8.b003
root@vm1:/home/joyo# ping 172.16.1.2
PING 172.16.1.2 (172.16.1.2) 56(84) bytes of data.
Layer 2 Unicast
Ingress traffic triggers local learn
Pod1 Pod2
EP1
172.16.1.1/24 a-leaf101# show endpoint mac 0050.56a8.b003 detail | grep epg-l2-2
0050.56a8.b003 123/CiscoLive2020:vrf1 vlan-1011 0050.56a8.b003 L eth1/26 CiscoLive2020:ap1:epg-l2-2
Layer 2 Unicast
Ingress leaf updates COOP record on Spines
a-spine1# show coop internal info repo ep | grep -B 8 -A 35 00:50:56:A8:B0:03

Spine Spine ------------------------------------------
**ommitted
EP bd vnid : 15761417
Pod1 EP mac : 00:50:56:A8:B0:03
**ommitted
Tunnel nh : 10.0.72.67
**omitted
Who owns the tunnel next-hop?

Leaf Leaf Leaf a-apic1# moquery -c ipv4Addr -f ‘ipv4.Addr.addr==“10.0.72.67”’
Total Objects shown: 1
# ipv4.Addr
EP1 addr : 10.0.72.67/32
172.16.1.1/24 dn : topology/pod-1/node-101/sys/ipv4/inst/dom-overlay-1/if-[lo0]/addr-[10.0.72.67/32]
0050.56a8.b003 **ommitted
Leaf 101 has the local learn
Layer 2 Unicast
How does the remote pod learn about the EP?
Local Pod Spine installs COOP record

show coop internal info repo ep | grep -B 8 -A 35 <mac address>
Local Pod Spine Exports into BGP EVPN

show bgp l2vpn evpn <mac address> vrf overlay-1
Remote Pod Spine Receives through EVPN

show bgp l2vpn evpn <mac address> vrf overlay-1
Remote Pod Spine Imports into COOP

show coop internal info repo ep | grep -B 8 -A 35 <mac address>
Layer 2 Unicast
Local spines exports to evpn
EVPN a-spine1# show bgp l2vpn evpn 00:50:56:A8:B0:03 vrf overlay-1

Spine Spine Route Distinguisher: 1:16777199 (L2VNI 1)
BGP routing table entry for
[2]:[0]:[15761417]:[48]:[0050.56a8.b003]:[0]:[0.0.0.0]/216, **ommitted
Pod1 Paths: (1 available, best #1)
COOP Flags: (0x00010a 00000000) on xmit-list, is not in rib/evpn
Multipath: eBGP iBGP
Advertised path-id 1
Path type: local 0x4000008c 0x0 ref 0, path is valid, is best path
AS-Path: NONE, path locally originated
Leaf Leaf Leaf 0.0.0.0 (metric 0) from 0.0.0.0 (192.168.1.101) Originated Locally
Origin IGP, MED not set, localpref 100, weight 32768
Received label 15761417 BD VNID
Extcommunity:
EP1 RT:5:16
172.16.1.1/24
0050.56a8.b003 Path-id 1 advertised to peers: Advertised to
192.168.2.101 192.168.2.102 Remote Pod Spines
Layer 2 Unicast
Remote spines receive EP through EVPN
a-spine3# show bgp l2vpn evpn 00:50:56:A8:B0:03 vrf overlay-1

Route Distinguisher: 1:16777199
BGP routing table entry for [2]:[0]:[15335345]:[48]:[0050.56a8.b003]:[0]:[0.0.0.0]/216, *ommitted
Paths: (2 available, best #1)
Flags: (0x000202 00000000) on xmit-list, is not in rib/evpn, is locked
Multipath: eBGP iBGP
Dataplane BGP Address
Advertised path-id 1
TEP/ETEP of POD1 of Spine 1
Path type: internal 0x40000018 0x2040 ref 1, path is valid, is best path
AS-Path: NONE, path sourced internal to AS
192.168.1.254 (metric 3) from 192.168.1.101 (192.168.1.101)
Origin IGP, MED not set, localpref 100, weight 0
Received label 15335345
Received path-id 1
Extcommunity:
RT:5:16
ENCAP:8
Layer 2 Unicast
What is the Dataplane TEP/External Proxy TEP (ETEP)?
• Per-Pod anycast address a-apic1# moquery -c ipv4If -f 'ipv4.If.mode*"etep"' -x 'rsp-subtree=children'
• Owned by all spines in a # ipv4.If

id : lo14 Loopback14 on Node 1001
pod adminSt
dn
: enabled
: topology/pod-1/node-1001/sys/ipv4/inst/dom-overlay-1/if-[lo14]
donorIf : unspecified
• Set as Next Hop for BGP lcOwn : local
EVPN Paths modTs
mode
: 2019-02-20T16:58:34.113-04:00
: etep
rn : if-[lo14] Multipod Dataplane TEP/ETEP
• COOP Placeholder for
# ipv4.Addr
external proxy lookups addr : 192.168.1.254/32 NH Address in BGP Update
**ommitted
Not Actually Used for Dataplane!
Layer 2 ETEP Lookup
Spine COOP Forward to
Spine
Proxied Layer Lookup Points Remote Pod
2 Traffic to Remote External MAC
POD ETEP Proxy TEP
apic1# moquery -c ipv4If -f 'ipv4.If.mode=="anycast-mac,external"' -x 'rsp-subtree=children' | egrep "addr" | grep dn

dn : topology/pod-1/node-1001/sys/ipv4/inst/dom-overlay-1/if-[lo8]/addr-[10.0.0.33/32]
Layer 3 ETEP Lookup

Spine Spine COOP Forward to
Proxied Layer Lookup Points Remote Pod
3 Traffic to Remote External v4/v6
POD ETEP Proxy TEP
apic1# moquery -c ipv4If -f 'ipv4.If.mode=="anycast-v4,external"' -x 'rsp-subtree=children' | egrep "addr" | grep dn

Layer 2 Unicast
Verify Remote Pod COOP Entry
a-spine3# show coop internal info repo ep | grep -B 8 -A 35 00:50:56:A8:B0:03 Spine Spine
------------------------------------------
**ommitted Pod2
EP bd vnid : 15761417 Proxied L2 Traffic will forward
EP mac : 00:50:56:A8:B0:03 to the Pod1 External MAC-
Remote Type : MPOD proxy Address
MAC Tunnel : 10.0.0.33
IPv4 Tunnel : 10.0.0.34
IPv6 Tunnel : 10.0.0.35
ETEP Tunnel : 192.168.1.254 Leaf
**ommitted
Leaf Leaf
COOP Entry exists and points to POD1
Layer 2 Unicast
BD Settings - UUC Proxy, ARP Flooding Enabled, UC Routing Disabled
3 Remote Spines have
2 Local Spines have COOP entry pointing to pod1-leaf101# show endpoint mac 8c60.4f02.88fc
COOP entry pointing Local Pod Leaf <no output>
to remote ETEP
Pod1 Pod2 pod1-leaf101# show isis dteps vrf overlay-1

10.0.120.33 SPINE PHYSICAL,PROXY-ACAST-MAC
1 No EP entry for
dmac, send to proxy Local POD Leaf receives,
4 installs tunnel to source
leaf, and forwards to EP
EP1 root@vm1:/home/joyo# arp -an EP2

172.16.1.1/24 ? (172.16.1.2) at 8c:60:4f:02:88:fc [ether] on ens192 172.16.1.2/24
0050.56a8.b003 8c60.4f02.88fc
Dynamic Tunnel Learns
Vxlan Tunnels are Created 3 Ways
a-leaf205# moquery -c tunnelIf -f 'tunnel.If.id=="tunnel1"'
id : tunnel1
dest : 10.0.72.67
Remote Pod Endpoint Learns idRequestorDn : sys/inst-overlay-1/db-dtep/dtep-[10.0.72.67]
id : tunnel1
Remote POD dest : 10.0.72.64
idRequestorDn : sys/bgp/inst/dom-overlay-1/db-dtep/dtep-[10.0.72.64]
External Routes
# tunnel.If
id : tunnel1
Local POD ISIS dest : 10.0.152.64
idRequestorDn : sys/isis/inst-default/dom-overlay-1/lvl-l1/db-dtep/dtep-[10.0.152.64]
Database
Endpoint Created Tunnels
ping 172.16.1.2
Pod1 Pod2
TEP Pool: TEP Pool:
10.0.0.0/17 10.0.128.0/17
TEP: 10.0.72.67 TEP: 10.0.200.67
Leaf Leafs install white-list for remote Leaf
TEP ranges
EP1 EP1
172.16.1.1/24 172.16.1.2/24
0050.56a8.b003 8c60.4f02.88fc
Endpoint (Dataplane) Created Tunnels
Outer Dst IP Outer Src IP Inner Dst IP Inner Src IP
10.0.200.67 10.0.72.67 172.16.1.2 172.16.1.1
Verify White-List on Dst Leaf Verify Tunnel Created on Dst Leaf

vsh_lc -c "show sys internal eltmc info pfxwl_table" | grep "Prefix" | grep -v Tunnel moquery -c tunnelIf -f 'tunnel.If.dest=="10.0.72.67"'
**ommitted Total Objects shown: 1
Prefix: 10.0.20.64
Prefix len: 27 # tunnel.If
Prefix: 10.0.64.64 id : tunnel3
Prefix len: 27 dest : 10.0.72.67
Prefix: 10.0.72.64 Src TEP matches here operSt : up
Prefix len: 27 **ommitted
**ommitted
White-list Prefixes based on dhcp pools within remote POD TEP-Range
Layer 2 Unicast
Remote Leaf Installs EP to Source
a-leaf205# show endpoint mac 0050.56a8.b003 detail

**ommitted Spine Spine
100/CiscoLive2020:vrf1 0050.56a8.b003 tunnel16 CiscoLive2020:bd-L2-2
Pod2
Where does tunnel16 point?
a-leaf205# show interface tunnel16

Tunnel16 is up
Tunnel source 10.0.200.67/32 (lo0)
Tunnel destination 10.0.72.67 Leaf Leaf Leaf
**Ommitted
a-apic1# moquery -c ipv4Addr -f ‘ipv4.Addr.addr==“10.0.72.67”’
addr : 10.0.72.67/32
**ommitted
Layer 2 Unicast
Return Path…
2 Spines simply
provide transit
Pod1 Pod2
3 Pod1 Leaf installs
tunnel and remote
learn to pod 2 leaf 1 Pod2 Leaf Forwards
Based on Remote Learn
EP1 EP2
172.16.1.1/24 172.16.1.2/24
0050.56a8.b003 8c60.4f02.88fc
Using Ftriage to Troubleshoot Multipod (14.2+)
Look for bridged flow
*Recommended with EX or Later Hardware ingressing 101 or 103
Frame seen on
a-apic1# ftriage bridge -ii LEAF:101,103 -dip 172.16.1.2 -sip 172.16.1.1 leaf101
Starting ftriage
ftriage: main:839 L2 frame Seen on a-leaf101 Ingress: Eth1/30 (Po15) Egress: Eth1/54 Vnid: 16056274
ftriage: main:242 ingress encap string vlan-1062 Frame seen on spine2
ftriage: main:839 L2 frame Seen on a-spine2 Ingress: Eth1/25 Egress: Eth1/31 Vnid: 16056274
ftriage: fib:332 a-spine2: Transit in spine
ftriage: unicast:1458 a-spine2: Infra route 10.0.200.67 present in RIB
ftriage: unicast:1681 a-spine2: Packet is exiting the fabric through {a-spine2: ['Eth1/31']}
ftriage: main:839 L2 frame Seen on a-spine3 Ingress: Eth1/29 Egress: LC-1/3 FC-22/0 Port-1 Vnid: 16056274
ftriage: fib:332 a-spine3: Transit in spine
Frame seen on pod2 spine3
ftriage: unicast:1458 a-spine3: Infra route 10.0.200.67 present in RIB
ftriage: unicast:1774 L2 frame Seen on FC of node: a-spine3….
ftriage: main:622 Found peer-node a-leaf205 and IF: Eth1/53 in candidate list
ftriage: main:839 L2 frame Seen on a-leaf205 Ingress: Eth1/53 Egress: Eth1/31 Vnid: 11371
ftriage: main:522 Computed egress encap string vlan-1039
ftriage: main:332 Egress BD(s): jy:cl1 Frame seen on pod2 leaf205
ftriage: unicast:1833 a-leaf205: Dst EP is local
ftriage: misc:657 a-leaf205: EP if(Eth1/31) same as egr if(Eth1/31) Forwards out eth1/31
Troubleshooting Scenario: Pod 1 Verifications
EP’s cannot communicate in L2 BD
1 Is communication unicast or multi-destination?

root@vm1:/home/joyo# arp -an
? (172.16.1.2) at 8c:60:4f:02:88:fc [ether] on ens192 ARP is resolved so host sends unicast
a-leaf101# show endpoint mac 8c60.4f02.88fc Ingress leaf has no remote learn
<no entry>
Does BD flood or proxy unknown unicast?

a-apic1# moquery -c fvBD -f 'fv.BD.name=="bd-L2-2“’
name : bd-L2-2
dn : uni/tn-CiscoLive2020/BD-bd-L2-2
unkMacUcastAct : proxy UUC set to “Proxy”
2 Does Local Pod Spine have the EP?

a-spine1# moquery -c coopEpRec -f 'coop.EpRec.mac=="8c60.4f02.88fc "'
No Mos found MAC not in COOP
a-spine1# show bgp l2vpn evpn 8c60.4f02.88fc vrf overlay-1

<no output>
MAC not in EVPN
3 Does Remote Pod Spine have the EP?

a-spine3# moquery -c coopEpRec -f 'coop.EpRec.mac=="8c60.4f02.88fc "'
# coop.EpRec
vnid : 15761417 MAC is in COOP
mac : 8C:60:4F:02:88:FC
**ommitted**
a-spine3# show bgp l2vpn evpn 8c60.4f02.88fc vrf overlay-1

<ommitted>
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.2.101)
Origin IGP, MED not set, localpref 100, weight 32768 Remote Spine
Received label 15761417 Exports to EVPN
Extcommunity:
RT:5:16
Troubleshooting Scenario: Pod 1 or Pod2
Verifications
4 Is EVPN up between Pods?
a-spine1# show bgp l2vpn evpn summ vrf overlay-1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

192.168.2.101 4 65000 57380 66362 0 0 0 00:00:21 Active BGP is down
192.168.2.102 4 65000 57568 66357 0 0 0 00:00:22 Active
Next Steps…
• Do the local spines have routes
to remote spines?
• Does IPN support jumbo MTU?
• Can spines ping between each
other?
Layer 3 Unicast
…nearly identical to layer 2 unicast
• Differences from Layer 2

• VRF Lookup rather than BD lookup
• VRF VNID used instead of BD VNID
• Spines trigger ARP Glean if Dst is Unknown (leverages fabric multicast)
Layer 3 Unicast – Glean Scenario
BD Settings - UC Routing Enabled
Next-hop is spine
Proxy
3a-leaf101# show isis dtep vrf overlay-1 | grep 10.0.120.34
Spine Spine
10.0.120.34 SPINE N/A PHYSICAL,PROXY-ACAST-V4
Pod1 2a-leaf101# show ip route 172.16.2.2 vrf CiscoLive2020:vrf1 Pervasive Flag

indicates BD Subnet
172.16.2.0/24, ubest/mbest: 1/0, attached, direct, pervasive Static Route
*via 10.0.120.34%overlay-1, [1/0], 12:02:55, static
recursive next hop: 10.0.120.34/32%overlay-1
No EP learn, check
1a-leaf101# show endpoint ip 172.16.2.2 routing table
Leaf Leaf Leaf +--------+-------+-------------+-----------+-----------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+--------+-------+-------------+-----------+-----------+
EP1
172.16.1.1/24 root@vm1:/home/joyo# ping 172.16.2.2
0050.56a8.b003 PING 172.16.2.2 (172.16.2.2) 56(84) bytes of data.
Layer 3 Unicast – Glean Scenario No COOP Entry! This
will trigger a Glean
BD Settings - UC Routing Enabled
Local Spines have no a-spine1# show coop internal info ip-db | grep -F -B 1 -A 15 “172.16.2.2"
COOP entry for Dst IP
Pod1 Pod2
172.16.2.2 not
Leaf Leaf Leaf Leaf Leaf Leaf learned yet
EP1 EP2
172.16.1.1/24 root@vm1:/home/joyo# ping 172.16.2.2 172.16.2.2/24
0050.56a8.b003 PING 172.16.2.2 (172.16.2.2) 56(84) bytes of data. 8c60.4f02.88fc
Layer 3 Unicast
What is a Glean?
• If the Spines do not have an IP learn

and…
• The destination IP is within a deployed BD Subnet
✓ The spine floods the proxied request with a special ethertype
✓ Gleans flooded to 239.255.255.240 (*see hidden slide)
✓ Leafs with destination subnet generate ARP
Inter-Pod Glean ERSPAN of Spine > IPN Link
Custom Ethertype
for Gleans
nexus5k# ping 172.16.2.2 source 172.16.1.1 vrf jy1

PING 172.16.2.2 (172.16.2.2) from 172.16.1.1: 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out Src: Originating Leaf
Dst: Reserved Glean
Multicast Group
Source IP that triggered Glean:

0xac100101 = 172.16.1.1
Dst IP that triggered Glean:

0xac100202 = 172.16.2.2
System Gipo Usage
• If “Use Infra as System Gipo” is enabled actual BD gipo’s used
rather than 239.255.255.240
Capturing a Glean with Tcpdump
ACI Leafs and Spines contain pseudo interfaces for traffic to and from the CPU
• Traffic on the on the knet or tahoe pseudo interface will have a special ieth header. It must be decoded.
• Starting in 3.2 the knet_parser.py script is available on the switch cli to decode
1st Gen Leaf • For traffic going to the cpu

check knet0 and kpm_inb
Phys knet0 CPU
ASIC • For traffic coming from the
Port knet1 kpm_inb
cpu check knet1 and
kpm_inb
EX (or Later) Leaf

• For traffic to and from the
Phys CPU cpu check Tahoe0 and
ASIC Tahoe0
Port kpm_inb kpm_inb
*Note, not all traffic will show up on the kpm_inb interface. However, all
traffic shows on the pseudo interface
*Gen1 and 2 Modular spines use psdev0, psdev1, and psdev2 interfaces.
Gen 2 fixed spines use tahoe0. Gen 1 fixed spines use knet0-3
Egress Leaf Verification
Gen2 or Later Leaf
tcpdump -xxxvei tahoe0 -w /bootflash/tahoe0.pcap Decode type
knet_parser.py --file /bootflash/tahoe0.pcap --pcap --decoder tahoe should be tahoe for
tahoe interface
Frame 111 RX sup traffic
Time: 2019-05-16T16:56:33.059831+00:00 rather than TX
Header: ieth_extn CPU Receive
sup_qnum:0x14, sup_code:0x21, istack:ISTACK_SUP_CODE_SPINE_GLEAN(0x21)
Header: ieth
sup_tx:0, ttl_bypass:0, opcode:0x6, bd:0x120e, outer_bd:0x27, dl:0, span:0, traceroute:0, tclass:0
src_idx:0x3a, src_chip:0x0, src_port:0x19, src_is_tunnel:1, src_is_peer:1
dst_idx:0x0, dst_chip:0x0, dst_port:0x0, dst_is_tunnel:0
Len: 148
Eth: 000d.0d0d.0d0d > 0100.5e7f.fff1, len/ethertype:0x8100(802.1q)
802.1q: vlan:2, cos:5, len/ethertype:0x800(ipv4)
ipv4: 10.0.116.64 > 239.255.255.241, len:130, ttl:249, id:0x0, df:0, mf:0, offset:0x0, dscp:32, prot:17(udp)
udp: (ivxlan) 0 > 48879, len:110 Switch recognizes
ivxlan: n:1, l:1, i:1, this as a Glean
vnid: 0x2b0000 Traffic that
lb:0, dl:1, exception:0, src_policy:0, dst_policy:0, src_class:0x5c0 triggered Glean
mcast(routed:0, ingress_encap:0/802.1q), ac_bank:0, src_port:0x0
Eth: 000c.0c0c.0c0c > ffff.ffff.ffff, len/ethertype:0xfff2(aci-glean)
ipv4: 172.16.1.1 > 172.16.2.2, len:84, ttl:63, id:0x71f9, df:1, mf:0, offset:0x0, dscp:0, prot:1(icmp)
icmp: echo request id:0x9092, seq:0x1980
Egress Leaf Verification
Gen1 Leaf Example
knet0 would show Rx traffic (similar output as Tahoe0)
tcpdump -xxxvei knet0 -w /bootflash/knet0.pcap
knet_parser.py --file /bootflash/knet0.pcap --pcap --decoder knet
knet1 would show Tx traffic

tcpdump -xxxvei knet1 -w /bootflash/knet1.pcap
knet_parser.py --file /bootflash/knet1.pcap --pcap --decoder knet
No decode necessary for kpm_inb (cpu) interface…Gleans aren’t easily readable

tcpdump -xxxvei kpm_inb ether proto 0xfff2
a-leaf102# tcpdump -xxxvei kpm_inb ether proto 0xfff2
tcpdump: listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes
15:27:37.663580 00:0c:0c:0c:0c:0c (oui Unknown) > Broadcast, ethertype Unknown (0xfff2), length 94:
0x0000: ffff ffff ffff 000c 0c0c 0c0c fff2 4500
0x0010: 0054 aa4b 4000 3f01 825d 0404 0464 0303
0x0020: 0396 0800 0dc6 2384 38db 5275 dd5c 0000
0x0030: 0000 9e35 0100 0000 0000 1011 1213 1415
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233
Layer 3 Unicast – Glean Scenario
IPN Must Route 239.255.255.240 (*see Troubleshooting Multidestination Flows Section)
IPN1# show run | grep 239

ip pim rp-address 192.168.100.1 group-list 239.0.0.0/8 bidir
ip pim rp-address 10.10.1.1 group-list 239.0.0.0/8 bidir
Verify ARP on Remote Leaf

Endpoint
a-leaf205#show ip arp internal event-history event | grep -F -B 1 172.16.2.2 Learn Installed
73) Event:E_DEBUG_DSF, length:127, at 316928 usecs after Wed May 1 08:31:53 2019
Updating epm ifidx: 1a01e000 vlan: 105 ip: 172.16.2.2, ifMode: 128 mac: 8c60.4f02.88fc Response
75) Event:E_DEBUG_DSF, length:152, at 316420 usecs after Wed May 1 08:31:53 2019 Received
log_collect_arp_pkt; sip = 172.16.2.2; dip = 172.16.2.254; interface = Vlan104;info = Garp Check adj:(nil) ARP Request is
77) Event:E_DEBUG_DSF, length:142, at 131918 usecs after Wed May 1 08:28:36 2019 generated by leaf
log_collect_arp_pkt; dip = 172.16.2.2; interface = Vlan104;iod = 138; Info = Internal Request Done
Glean
Glean Received,
Group Range Dst IP
log_collect_arp_glean;dip = 172.16.2.2;interface = Vlan104;info = Received pkt Fabric-Glean: 1
included
is inasBD
Bidir
Subnet
on IPN
log_collect_arp_glean; dip = 172.16.2.2; interface = Vlan104; vrf = CiscoLive2020:vrf1; info = Address in PSVI subnet or special VIP
Look for routed flow

L3 Proxy/Glean Scenario ingressing 101 or 103 Frame seen on
leaf103
a-apic1# ftriage route -ii LEAF:101,103 -dip 172.16.2.3 -sip 172.16.1.1
ftriage: main:839 L3 packet Seen on a-leaf103 Ingress: Eth1/30 (Po15) Egress: Eth1/49 Vnid: 2588674
ftriage: main:242 ingress encap string vlan-1062
ftriage: main:301 Ingress Ctx: jy:vrf11
ftriage: main:933 SIP 172.16.1.1 DIP 172.16.2.3 Dst Unknown, Proxy!
ftriage: unicast:973 a-leaf103: <- is ingress node
Seen on spine1
ftriage: unicast:1194 a-leaf103: Dst EP is unknown - proxy
ftriage: main:839 L3 packet Seen on a-spine1 Ingress: Eth2/29 Egress: LC-2/3 FC-23/0 Port-1 Vnid: 2588674
ftriage: fib:323 a-spine1: EP not found in COOP! for VRF VNID: 2588674
ftriage: unicast:1373 a-spine1: EP is unknown in COOP. Ftriage will exit but continue with further fault isolation
ftriage: unicast:1412 a-spine1: Egress node not provided. Cannot check local EP. Exiting!
ftriage: unicast:1413 : Ftriage Completed with hunch: Check if local EP learnt on egress node(s)
EP not in COOP!
✓ EP Not in COOP, gleans should be generated.
Check local learn on egress leaf
Troubleshooting
Multidestination
Flows
Multipod Multicast
What does Multipod use BUM for?
• Unknown Unicast Flooding

• Multidestination Traffic (ARP, Multicast, BPDU’s)
• Inter-pod Glean Messages
• EP Announce Messages
…it doesn’t just affect multidestination flows
IPN Multicast Control-plane
• Spines act has multicast hosts (IGMP only)
• Spines join fabric multicast groups (Gipo’s)
• IPN’s receive Joins
• IPN’s send PIM joins to RP
• PIM Bidir is used so no (S,G)
What is a Gipo?
• Multicast group allocated per-VRF and per-BD.

• Used for all flooded traffic
a-apic1# moquery -c fvBD -f 'fv.BD.name=="bd-L3-1"'
# fv.BD
name : bd-L3-1
bcastP : 225.0.80.64
dn : uni/tn-CiscoLive2020/BD-bd-L3-1
ipLearning : yes
multiDstPktAct : bd-flood
unicastRoute : yes
unkMacUcastAct : proxy
unkMcastAct : flood
v6unkMcastAct : flood
IPN
Spine sends IGMP

Join for GIPO
RP
IPN Routers Send

Spine Spine PIM Join to RP Spine Spine
Pod1 Pod2
BD Gipo Ex: BD Gipo Ex:

225.0.80.64 225.0.80.64
IPN Multicast Dataplane
IPN
All Multicast Dataplane
Goes Through RP
RP
Pod1 Pod2

225.0.80.64 225.0.80.64
Only one spine in each pod joins each group
Pod1 Pod2
IPN IPN IGMP Join
IGMP Join
225.0.80.64 225.0.80.64
a-spine1# show ip igmp gipo joins

GIPo list as read from IGMP-IF group-linked list
------------------------------------------------
225.0.80.64 0.0.0.0 Join Eth1/25.25 95 Enabled
Spine1 Joins for Pod1
Only one spine in each pod joins each group
Pod1 Pod2
IPN IPN IGMP Join
IGMP Join
225.0.80.64 225.0.80.64
IPN1# show ip mroute 225.0.80.64 vrf IPN IPN1# show ip igmp groups 225.0.80.64 vrf IPN
IP Multicast Routing Table for VRF “IPN"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
(*, 225.0.80.64/32), bidir, uptime: 13:00:48, igmp ip pim Group Address Interface Uptime Expires Last Reporter
Incoming interface: loopback1, RPF nbr: 192.168.100.1 225.0.80.64 Ethernet1/1.4 13:02:14 00:04:02 192.168.1.0
Outgoing interface list: (count: 3)
Ethernet8/2, uptime: 01:34:42, pim
loopback1, uptime: 13:00:48, pim, (RPF)
Ethernet1/1.4, uptime: 13:00:48, igmp
RPF for all IPN’s must point to same RP
IPN1 IPN3
Pod1 IGMP Join IGMP Join Pod2
BD Gipo Ex: RPF BD Gipo Ex:

225.0.80.64 225.0.80.64
IPN2 RP
IPN3# show ip mroute 225.0.80.64 vrf IPN IPN3# show ip pim rp 225.0.80.64 vrf IPN
IP Multicast Routing Table for VRF "IPN" PIM RP Information for group 225.0.80.64 in VRF "IPN"
RP: 192.168.100.1, (1)
RPF must not
(*, 225.0.80.64/32), bidir, uptime: 01:34:35, igmp ip pim
point to ACI
Incoming interface: Ethernet8/25, RPF nbr: 10.255.0.0 IPN3# show ip route 192.168.100.1 vrf IPN
Outgoing interface list: (count: 2) 192.168.100.0/30, ubest/mbest: 1/0
Ethernet8/25, uptime: 01:34:35, pim, (RPF) *via 10.255.0.0, Eth8/25, [110/5], 13:01:42, ospf-IPN, intra
Ethernet1/17.4, uptime: 01:34:35, igmp
Phantom RP
• Bidir PIM doesn’t support multiple RP’s

• Phantom RP is only means of RP redundancy
• Works by advertising varied Prefix Lengths for RP subnet
• Failover handled via IGP
• Loopback must be OSPF P2P network type
• Exact RP Address must not exist anywhere
Phantom RP Load-Balancing
Requirement: Each multicast group must have only one RP

To use multiple RP’s…
Break mcast group range into smaller groups
225.0.0.0/8 becomes
225.0.0.0/9 – RP 192.168.255.1
225.128.0.0/9 – RP 192.168.255.2
Phantom RP
RP Addr - 192.168.255.1
IPN1# show run int lo1 IPN3# show run int lo1
interface loopback1 IPN1 IPN3 interface loopback1

ip address 192.168.255.2/27 ip address 192.168.255.2/28
ip ospf network point-to-point ip ospf network point-to-point
ip router ospf IPN area 0.0.0.0 ip router ospf IPN area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode
IPN2# show run int lo1 IPN2 IPN4 IPN4# show run int lo1
interface loopback1 interface loopback1

ip address 192.168.255.2/29 ip address 192.168.255.2/30
ip ospf network point-to-point ip ospf network point-to-point
ip router ospf IPN area 0.0.0.0 ip router ospf IPN area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode
IPN4 is RP due to Longest Prefix
Common Multicast Problems
Issue #1: RP Address Exists on Multiple Routers
IPN1# show run int lo1 RP Addr - 192.168.255.1

interface loopback1 IPN1 IPN1# show ip route 192.168.255.1 vrf IPN
ip address 192.168.255.1/29 IP Route Table for VRF "IPN"
ip router ospf IPN area 0.0.0.0 192.168.100.1/32, ubest/mbest: 1/0, attached
ip pim sparse-mode *via 192.168.100.1, Lo1, [0/0], 21:01:48, local
IPN2# show run int lo1 IPN2

• IPN Routers see local /32 for RP
interface loopback1
ip address 192.168.255.1/30
address
ip router ospf IPN area 0.0.0.0 • All Routers think they are RP
ip pim sparse-mode
Exact RP address can’t exist with Phantom RP
Issue #2: RP Loopback not OSPF P2P Network
IPN1# show run int lo1 RP Addr - 192.168.255.1

interface loopback1 IPN1 IPN1# show ip route 192.168.255.1 vrf IPN
ip address 192.168.255.2/29 IP Route Table for VRF "IPN"
ip router ospf IPN area 0.0.0.0
ip pim sparse-mode 192.168.100.0/29, ubest/mbest: 1/0, attached
*via 192.168.100.2, Lo1, [0/0], 21:15:36, direct
IPN2# show run int lo1 IPN2
interface loopback1 Where is the /30 from IPN2?

ip address 192.168.255.2/30
ip router ospf IPN area 0.0.0.0
ip pim sparse-mode
Issue #2: RP Loopback not OSPF P2P Network
• Loopbacks advertise /32 by default
Without P2P Network Type With P2P Network Type

IPN1#show ip ospf database router 1.1.1.1 detail vrf IPN IPN1# show ip ospf database router 1.1.1.1 detail vrf IPN
Link connected to: a Stub Network Link connected to: a Stub Network
(Link ID) Network/Subnet Number: 192.168.255.2 (Link ID) Network/Subnet Number: 192.168.255.0
(Link Data) Network Mask: 255.255.255.255 (Link Data) Network Mask: 255.255.255.248
Number of TOS metrics: 0 Number of TOS metrics: 0
TOS 0 Metric: 1 TOS 0 Metric: 1
/32 Advertised /29 Advertised
Configure “ip ospf network point-to-point” on RP loopbacks
Issue #3: RPF Points to ACI
IPN3# show ip mroute 225.0.80.64 vrf IPN
(*, 225.0.80.64/32), bidir, uptime: 00:00:26, igmp ip pim

High Speed Link: Cost 1 IPN1
IPN IPN3 Incoming interface: Ethernet1/1.4, RPF nbr: 192.168.1.0
Outgoing interface list: (count: 2)
Ethernet1/1.4, uptime: 00:00:26, igmp, (RPF)
Low Speed Link: Cost 10
IPN2 RP
Spine Spine
Pod1 Pod2
• Spines don’t run PIM so not valid RPF

• Applicable when single spine is connected to multiple IPN’s
RPF is Eth1/1.4 (ACI) No PIM Neighbor on that Link

IPN3# show ip mroute 225.0.80.64 vrf IPN IPN1# show ip pim int eth1/1.4 brief vrf IPN
PIM Interface Status for VRF “IPN"
(*, 225.0.80.64/32), bidir, uptime: 00:00:26, igmp ip pim Interface IP Address PIM DR Address Neighbor
Incoming interface: Ethernet1/1.4, RPF nbr: 192.168.1.0 Count
Outgoing interface list: (count: 2) Ethernet1/1.4 192.168.1.1 192.168.1.1 0
Ethernet1/1.4, uptime: 00:00:26, igmp, (RPF)
IPN1
IPN IPN3
High Speed Link: Cost 1
Make IPN-IPN links have
Low Speed Link: Cost 10
equal or better OSPF Cost
IPN2 RP
Spine Spine
Pod1 Pod2
Troubleshooting
External Routed
Communication
External Routed L3out Control-Plane
Almost the same as traditional L3outs
• External Routes redistributed into Fabric BGP

• Each pod is BGP VPNv4 Route-Reflector Cluster
• Spines reflect external routes across pods
• Internal Leafs import VPNv4 routes
• Internal Leafs see Border Leaf as next hop
Spines Reflect VPNv4
3 Paths between Pods
How do internal Leafs
Spine Spine Spine learn external routes?
Pod1 Pod2
Internal Leafs Import Border Leaf Exports

4 Routes from BGP into BGP and sends
2 to Spines
Leaf Leaf Leaf Leaf

Border Leaf Learns
1 External Routes
Internal Leaf install tunnel to Border

5 Leaf based on BGP Next Hop
10.13.13.13/32
External Route on Internal Leaf
a-leaf101# show bgp ipv4 unicast 10.13.13.13/32 vrf CiscoLive2020:vrf1

Next-hop, this is the
Advertised path-id 1, VPN AF advertised path-id 1
Border Leaf TEP
Path type: internal adv path ref 2, path is valid, is best path
Imported from 10.0.200.67:10:13.13.13.13/32
AS-Path: NONE, path sourced internal to AS
10.0.200.67 (metric 64) from 10.0.64.64 (192.168.1.102)
Origin incomplete, MED 5, localpref 100, weight 0
Received label 0
Imported Route-target Received path-id 2 Source route AD is 110
Extcommunity: (must be OSPF)
RT:65000:2818051
Vxlan Vnid used for COST:pre-bestpath:165:2415919104 BGP Route-Reflector
traffic using this route VNID:2818051 Cluster-list, one for
COST:pre-bestpath:162:110 each pod
Originator: 10.0.200.67 Cluster list: 192.168.1.102 192.168.2.254
Tunnel Built by BGP on Internal Leaf
a-leaf101# show ip route 10.13.13.13 vrf CiscoLive2020:vrf1
IP Route Table for VRF "CiscoLive2020:vrf1"
'*' denotes best ucast next-hop
10.13.13.13/32, ubest/mbest: 1/0

External Route *via 10.0.200.67%overlay-1, [200/5], 1d00h, bgp-65000, internal, tag 65000
Learned via BGP recursive next hop: 10.0.200.67/32%overlay-1
a-leaf101# moquery -c tunnelIf -f 'tunnel.If.dest*"10.0.200.67"'
# tunnel.If *Note, Tunnel could be created

id : tunnel47 by unrelated EP learn first
Tunnel Created dest : 10.0.200.67
idRequestorDn : sys/bgp/inst/dom-overlay-1/db-dtep/dtep-[10.0.200.67]
by BGP type : fabric-ext,physical
vrfName : overlay-1
a-leaf101# vsh
a-leaf101# show bgp internal event-history objstore | grep a00c843 Dest IP in hex
Initial BGP 2019 Apr 2 21:12:30 bgp 65000 [58156]: TID 58302: (0) OBJ: bgp_dtep_add: tep=a00c843
Tunnel Creation
How do Border Leafs forward to internal Leafs?
• Exactly the same as non-multipod…

• Bridge Domain Static Route Pushed to Border Leaf by Contract
• Border Leafs Redistributes (if configured) into external protocol
• External > Internal traffic hits EP learn or BD static (proxy) route
How do Border Leafs forward to internal Leafs?
a-leaf205# show ip route 10.1.1.1 vrf CiscoLive2020:vrf1

Spine Spine Spine IP Route Table for VRF "CiscoLive2020:vrf1"
'*' denotes best ucast next-hop
Pod1 Pod2
10.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.120.34%overlay-1, [1/0], 22:14:10, static
recursive next hop: 10.0.120.34/32%overlay-1
Leaf Leaf Leaf Leaf
EPG A Contract L3Out

BD Subnet 10.1.1.0/24
Troubleshooting TIP
When Troubleshooting Layer 3 Flows Always…
1) Check if there is an Endpoint Learn

If not then…
2) Check if there is a BD (pervasive) static route
If not then…
3) Check if there is an External Route
L3Out Scenario Look for routed flow
ingressing 101 or 103
Frame seen on
leaf103
a-apic1# ftriage route -ii LEAF:101,103 -dip 10.13.13.13 -sip 172.16.1.1
ftriage: main:839 L3 packet Seen on a-leaf103 Ingress: Eth1/30 (Po15) Egress: Eth1/50 Vnid: 2588674
ftriage: main:242 ingress encap string vlan-1062
ftriage: main:301 Ingress Ctx: jy:vrf11
ftriage: nxos:1404 a-leaf103: nxos matching rule id:4572 scope:63 filter:65535
ftriage: main:933 SIP 172.16.1.1 DIP 10.13.13.13 Dst is behind L3out
ftriage: unicast:1058 a-leaf103: Dst EP is a WAN EP Sends to this TEP
ftriage: unicast:1070 a-leaf103: Policy enforcement mode is ingress (leaf 205)
ftriage: unicast:1215 a-leaf103: Dst EP is remote
ftriage: misc:657 a-leaf103: RwDMAC DIPo(10.0.200.67) is one of dst TEPs ['10.0.200.67']
ftriage: main:839 L3 packet Seen on a-spine2 Ingress: Eth1/27 Egress: Eth1/31 Vnid: 2588674
ftriage: main:839 L3 packet Seen on a-spine3 Ingress: Eth1/29 Egress: LC-1/3 FC-26/0 Port-1 Vnid: 2588674
ftriage: main:839 L3 packet Seen on a-leaf205 Ingress: Eth1/53 Egress: Eth1/31 Vnid: Null
ftriage: pktrec:490 a-leaf205: Collecting transient losses snapshot for LC module: 1
ftriage: fib:169 a-leaf205: L3 out interface Ethernet1/31
ftriage: main:522 Computed egress encap string vlan-1055 Arrives on 205 and forwards
ftriage: main:313 Building egress BD(s), Ctx out l3out bgp on vlan 1055
ftriage: main:331 Egress Ctx jy:vrf11
ftriage: main:332 Egress BD(s): jy:vrf11:l3out-bgp:vlan-1055
Common Multipod L3out Problems
Issue #1: Asymmetric Routing with Active/Active Pods
Spine Spine
• Both Pods advertise same
Pod1 Pod2 BD Subnet
BD1 BD1 • External Device performs

10.1.1.0/24 10.1.1.0/24 stateful inspection
• Return traffic ingresses
Leaf Leaf Leaf Leaf different pod than where EP
EP sends
outbound traffic exists
• Traffic dropped
EP1
10.1.1.1/24
Return traffic goes to Pod2
and is dropped by FW
Issue #1: Asymmetric Routing with Active/Active Pods
Spine Spine Implement Host Route

Pod1 Pod2 Advertisement
BD1 BD1 • Pods advertise local /32 EP
10.1.1.0/24 10.1.1.0/24 information
• Requires GOLF or Host
Leaf Leaf Leaf Leaf Border Route Feature (HBR
in 4.0)
Pod 1 Advertises
10.1.1.1/32
EP1
10.1.1.1/24
Return traffic goes to

Pod 1
Issue #2: Stretched L3out VIP Failover
IPN Forwards GARP

Spine Spine
Two Common Problems Here
Pod1 Pod2 • Same encap vlan not
deployed for each vlan –
breaks flooded traffic
• IPN isn’t routing multicast
Leaf Leaf Leaf Leaf properly
New Active FW
Sends GARP
Pod 1 Leafs don’t see GARP,
still think local FW is active
Standby VIP
Active VIP Active VIP
10.2.2.2
10.2.2.2 Pod 2 Becomes Active 10.2.2.2
Which VNID and Gipo should the l3out use?

a-apic1# moquery -c fvIfConn -f 'fv.IfConn.dn*"uni/tn-CiscoLive2020/out-EIGRP/"' “EIGRP” is the name
Total Objects shown: 2 of the L3out
# fv.IfConn
bcastP : 225.1.188.208
dn : uni/epp/rtd-[uni/tn-CiscoLive2020/out-EIGRP/instP-defaultNet]/node-101/stpathatt-[shared-
5596-A-VPC]/conndef/conn-[vlan-1052]-[52.52.52.101/24]
extEncap : vxlan-15466402
# fv.IfConn
bcastP : 225.1.188.208
dn : uni/epp/rtd-[uni/tn-CiscoLive2020/out-EIGRP/instP-defaultNet]/node-205/stpathatt-[shared-
5596-A-VPC]/conndef/conn-[vlan-1052]-[52.52.52.103/24]
extEncap : vxlan-15466402
The same VNID and GIPO is extended to nodes 101 and 205
If there’s a problem check these things…

• Ensure an SVI is used for the l3out (no flooding for routed interfaces)
• Ensure the same vlan encap is used in each pod
• Ensure the IPN agrees on the tree for the GIPO
• Ensure a GARP is sent by external router
• Check if the GARP is sent with COS 6 (more on this later)
Quality of Service
ACI QoS Overview
Key Points
• Fabric QoS is based on COS and DEI bits in outer L2 header
• Incoming COS on non-fabric ports not preserved through fabric but…
• Incoming COS is written into outer DSCP so it can be preserved on egress
• Traffic is level 3 by default (COS 0 + DEI 0)
• COS 6 traffic from IPN may be mistreated (pre-4.0)
ACI QoS Overview
Inner Header iVXLAN Outer Header Fabric QOS
flags Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID DIP SIP 802.1Q SMAC DMAC
EPG UDP
COS Function Notes

Used for
3,4,5 APIC, SPAN, Control Plane SPAN=low Priority tracing flows
within the
Reserved, sent to CPU;
6 iTraceroute fabric.
actual iTraceroute is dscp 6 Reserved for
CPU generated
2 Level 1 traffic
User Traffic
1 Level 2
1 Priority Class
0 Level 3 (Default)
2+DEI Level 4 New in 4.0!
3+DEI Level 5 User Traffic
5+DEI Level 6 5 Priority Classes
ACI QoS Overview
Where is QoS Behavior Configured?
Configuration Where is it Configured? Function
Dot1p Preserve Global Access Policies Causes egress leaf to rewrite cos
to original value when forwarding
QoS Class App EPG, Contract, Subject Defines prioritization of traffic
through the fabric
Custom QoS App EPG Re-marks traffic based on
incoming COS or DSCP
Target DSCP (L3out) L3out, Contract, Subject Sets the DSCP value
DSCP Class-Cos Infra > Networking > Protocols Spines re-map QoS of traffic
Translation Policy going to and coming from IPN/ISN
ACI QoS – Preserve COS
Egress leaf
rewrites COS
based on DSCP
ACI Forwarding and QoS – Preserve COS
Layer 2 COS encoded into most significant 3 bits of DSCP
flags
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC VNID DSCP DIP SIP 802.1Q SMAC DMAC
EPG
Note: Incoming COS

Outer COS Value
and DSCP is not used
unless custom QoS matches the Level
policy is configured (Contract/EPG)
Configure Dot1p Preserve! The egress leaf will

look at the 3 MSB bits of the DSCP value to know
which COS value to use for packet rewrite
Pre-4.0 COS 6 Problem Fix? Configure “DSCP class-cos
translation policy for L3 traffic”
Last hop IPN router The spine will map the outer COS
writes COS based on value to a new DSCP class on
DSCP egress and map DSCP to COS in
…DSCP 48 = COS6 4 Datacenter interconnect ingress
(IPN, ISN)
DC1 treats 3
IP packet
packet as with DSCP 48
iTraceroute
5
Data Data
Center 1 Center 2
2
Leaf forwards frame
towards DC1 with
1 COS 0 and an outer
Frame with DSCP of 48
COS 6 set 0b110 000
DSCP – COS
Translation Policy
✓ COS 6 Problem
solved by using
DSCP – COS
Translation Policy
Incoming traffic from IPN

now classified on DSCP
and not COS
Note: DSCP-COS Translation Policy Will Negate Dot1p Preserve
After 4.0 Software…
• All devices trust DSCP Pre-4.0
markings set on ingress leaf Traceroute,
Datacenter Spine
• QoS class is derived from interconnect COS6 + not forwarded
DSCP 48 on egress leaf
DSCP (IPN, ISN)
due to COS6
• Spine rewrites COS
received from IPN based on After 4.0
DSCP
Datacenter Spine Whichever
• Traceroute is DSCP 6 so interconnect COS6 +
COS 6 + DSCP 48 is DSCP 48 class DSCP 48
(IPN, ISN) maps to
forwarded normally
✓ DSCP – COS
Translation Policy
Not Required
QoS CLI’s
show queueing interface ethernetx/y See per class Interface Stats

show queueing interface ethernetx/y detail  Same as above + control and policy classes
show system internal qos classes Check queueing behavior per class
moquery -c qospInfraDscpMap Check COS to DSCP translation policy on spine
moquery -c qosInstPol Verify if dot1P preserve is enabled (apic or switch)
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.
Continue your education
Demos in the Walk-in

Cisco campus self-paced labs
Meet the engineer

Related sessions
1:1 meetings
Thank you

ACI Troubleshooting Multipod

Uploaded by

Copyright:

Available Formats

You might also like

ACI Troubleshooting Multipod

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ACI Troubleshooting Multipod

Uploaded by

Copyright:

Available Formats

ACI Troubleshooting:

Joseph Young, ACI Technical Leader

EPG Endpoint Group TC Topology Change

• Effective Troubleshooting requires understanding…

Understand the “why”

APIC APIC APIC

• TEP reachability must be communicated

Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf

APIC APIC APIC

1. Configure Pod 1 (TEP pool, infra l3out)

L3 Interface used for OSPF peering

Anycast Address used for Pod x > Pod 1

Not needed for

✓ Verify OSPF is up between Pod1 Spines and IPN

✓ Ensure IPN is pre-provisioned for Pod 2 Connections

➢ Assign TEP pool to

L3 Interface used for OSPF

Anycast Address used for Pod x > Pod 2

Not needed for multipod,

Leave default, allows PODs to

Spine > IPN subnets

Spine Spine Spine Spine

Pod1 Pod2 1. Remote Pod

Leaf Leaf Leaf

Spine Spine Spine Spine

Leaf Leaf Leaf

Default Gateway, used

Spine Spine Spine Spine

0.0.0.0/0, ubest/mbest: 1/0

CONFIG 172.22.0.1 - - [18/Apr/2019:14:13:16 +0000] "GET

APIC APIC 200 OK Code. GET

Spine Spine Spine Spine DHCP

Spine Spine Spine Spine

APIC APIC APIC

Check your setup Parameters! Cluster configuration ...

• Spines share EP’s via BGP EVPN

Troubleshooting Single Pod and Multipod is Similar!

Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf

EP1 root@vm1:/home/joyo# arp -an

Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf

a-spine1# show coop internal info repo ep | grep -B 8 -A 35 00:50:56:A8:B0:03

Who owns the tunnel next-hop?

Leaf 101 has the local learn

Local Pod Spine installs COOP record

Local Pod Spine Exports into BGP EVPN

Remote Pod Spine Receives through EVPN

Remote Pod Spine Imports into COOP

EVPN a-spine1# show bgp l2vpn evpn 00:50:56:A8:B0:03 vrf overlay-1

a-spine3# show bgp l2vpn evpn 00:50:56:A8:B0:03 vrf overlay-1

• Per-Pod anycast address a-apic1# moquery -c ipv4If -f 'ipv4.If.mode*"etep"' -x 'rsp-subtree=children'

• Owned by all spines in a # ipv4.If

Not Actually Used for Dataplane!

apic1# moquery -c ipv4If -f 'ipv4.If.mode=="anycast-mac,external"' -x 'rsp-subtree=children' | egrep "addr" | grep dn

Layer 3 ETEP Lookup

apic1# moquery -c ipv4If -f 'ipv4.If.mode=="anycast-v4,external"' -x 'rsp-subtree=children' | egrep "addr" | grep dn

COOP Entry exists and points to POD1