Professional Documents
Culture Documents
DGTL Brkaci 2934
DGTL Brkaci 2934
#CiscoLive
Agenda
• Introduction
• Multipod Overview
• Troubleshooting the Multipod Setup Process
• Troubleshooting Unicast Flows
• Troubleshooting Multi-destination Flows
• Troubleshooting External Routed Communication
• Quality of Service
• Conclusion
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Acronyms/Definitions
Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure MDT Multicast Distribution Tree
ACL Access Control List MST Multiple Spanning Tree
APIC/IFC Application Policy Infrastructure OSPF Open Shortest Path First Protocol
Controller/ Insieme Fabric Controller
BD Bridge Domain pcTag Policy Control Tag
COOP Council of Oracle Protocol PIM Protocol Independent Multicast
ECMP Equal Cost Multipath PL Physical Local
EP Endpoint SVI Switch Virtual Interface
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Multipod Overview
Feature Evolution
I said no marketing…why is this necessary?
• Effective Troubleshooting requires understanding…
• Why does the feature exist?
• What problems does it solve?
• How does it solve them?
• How do the components interact?
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Feature Evolution – Classic ACI
Spine Spine
• VXLAN TEP reachability
Single ISIS
learned through ISIS
Fabric COOP
MPBGP • Endpoint Repo on Spines
handled by COOP
• MP-BGP to distribute
Leaf Leaf Leaf Leaf external routes through fabric
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Feature Evolution
What if ACI must be extended to other locations?
• TEP reachability must be communicated
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Feature Evolution – Stretched Fabric
Spine Spine Spine Spine
Single
Fabric
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Feature Evolution – Stretched Fabric
• Transit leafs connect to all spines
• COOP, ISIS, and BGP extended across locations
Not scalable
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Feature Evolution – Multipod
IPN
P OSPF,PIM OSPF
PF , IG M , IG M • Single Fabric Extended
OS P
BGP VPNv4/EVPN • Each pod is local
Spine Spine Spine Spine
instance of ISIS and
ISIS Pod2 ISIS COOP
Pod1
COOP COOP
• Inter-pod connectivity
MPBGP MPBGP
through IPN
• Inter-pod BUM uses
Leaf Leaf Leaf Leaf Leaf Leaf PIM-Bidir
• BGP between pods to
share endpoints and
external routes
APIC APIC APIC
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
IPN Requirements
qOSPF
qDHCP relay
qJumbo MTU (9150 Bytes)
qRouted Subinterfaces
qPIM Bidir with at least /15 Mask
qQoS (optional)
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Troubleshooting
Multipod Setup
Multipod Setup Overview
1. Configure Pod 1 (TEP pool, infra l3out)
2. Configure Remote Pod (TEP pool, infra l3out)
3. Register Remote Pod Spines (DHCP)
4. Discover Remote Pod Leafs (LLDP)
5. Remote Pod APIC’s join cluster
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Multipod Setup Process
Setting up Pod 1 (Seed Pod)
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Multipod Setup Process
Ø Configure
Addressing for
Pod 1 Spine > IPN
connection
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Multipod Setup Process
Ø Configure OSPF
parameters for
Pod 1 Spine > IPN
connection
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Multipod Setup Process
Ø Configure
Dataplane TEP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Multipod Setup Process
Ø Review POD1
configurations
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Multipod Setup Process
After setting up Seed Pod (Pod 1)…
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Multipod Setup Process
Setting up Pod 2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Multipod Setup Process
Ø Configure OSPF
parameters
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Multipod Setup Process
Ø Configure
Dataplane TEP for
Pod 2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Where to find the less-known MPOD configurations?
Dataplane TEPs
from Setup
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Multipod Setup Process
Ø POD 2 Spines
should now be
discoverable
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
IPN
Remote Pod Discovery
DISCOVER
APIC APIC
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
IPN Remote Pod Discovery
Pod1 Pod2
2. IP Address from
Multipod l3out is
assigned.
OFFER
APIC APIC
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
What’s in the
DHCP OFFER?
• IP address from L3out Pod2 Facing IPN
IP address (relay)
interface profile
assigned
• Gateway is next-hop
for default route
Offered IP (From l3out
• Bootstrap file interface profile)
communicates
location of l3out Directory on APIC from which Spine downloads full
Config l3out configuration *full directory is
/firmware/fwrepos/fwrepo/boot/bootstrap-202.xml
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
IPN
Remote Pod Discovery
Pod1 Pod2
3. Spine configures
static default route
for APIC reachability
with NH of IPN.
Leaf Leaf Leaf pod2-spine2# vsh -c "show ip route 0.0.0.0/0 vrf overlay-1"
IP Route Table for VRF "overlay-1"
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
IPN
Remote Pod Discovery
switch# moquery -c topSystem
# top.System
address : 0.0.0.0
Spine Spine Spine Spine
bootstrapState : downloading-bootstrap-config
role : spine
Pod1 Pod2
4. Spine downloads
bootstrap XML from
APIC which contains
l3out configuration
Leaf Leaf Leaf
pod1-apic1# grep bootstrap /var/log/dme/log/access.log
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
IPN
Remote Pod Discovery
Lo0
DISCOVER
Spine Spine Spine Spine
Pod1 Pod2
5. Spine acts as self
relay for TEP DHCP
request
6. TEP address from
Leaf Leaf Leaf
POD2 pool is
assigned
OFFER
APIC APIC
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
IPN
Remote Pod Discovery
LLDP
Pod1 Pod2
7. Pod2 Leafs
discovered through
normal process
(LLDP/DHCP)
Leaf Leaf Leaf Leaf Leaf Leaf
APIC APIC
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
IPN
Remote Pod Discovery
Pod1 Pod2
8. Pod2 APIC(s) join
cluster
*Non-seed pod APICs
still use Pod1 TEP Pool!
Leaf Leaf Leaf Leaf Leaf Leaf
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Common Multipod Discovery Problems
Issue #1: Pod2 Spines Don’t Receive L3out IP or Config
Possible Causes
1. DHCP Relays on IPN point to APIC OOB rather than infra
üConfigure Relays to point to infra (show controller on APICs)
2. IPN doesn’t have route to APICs
üCheck that OSPF is up between IPN and Pod1
3. Miscabling results in Spine receiving IP in different subnet than GW
üCorrect cabling or addressing then remove and rediscover Spine
4. Spines can’t resolve ARP for connected IPN interface
üEnsure SW version supports multipod + spine hw (ex: for 9364C MPOD
supported in 3.1(1))
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Common Multipod Discovery Problems
Issue #2: Pod2 Spines Don’t Receive TEP Addresses
Ensure leafs are connected to spine
-Spine TEP not assigned until leaf-facing interfaces “up”
Ensure Leaf–facing
interfaces are “up”
so Spine gets TEP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Common Multipod Discovery Problems
Issue #3: Remote Pod APIC Not Joining Cluster
Check your setup Parameters! Cluster configuration ...
Enter the fabric name [ACI Fabric1]: CL-Fab
Enter the fabric ID (1-128) [1]: 1
Enter the number of active controllers in the fabric (1-9) [3]:
Ensure Pod ID is Enter the number of active controllers in the fabric (1-9) [3]: 3
Correct Enter the POD ID (1-12) [1]: 2
Is this a standby controller? [NO]:
Is this an APIC-X? [NO]:
Remote Pod APIC must Enter the controller ID (1-3) [1]: 3
use Pod 1 TEP Pool Enter the controller name [apic3]: p2-apic3
Enter address pool for TEP addresses [10.0.0.0/16]:
Note: The infra VLAN ID should not be used elsewhere in your
environment
ü Run “acidiag avread” to check setup config and should not overlap with any other reserved VLANs on other
platforms.
ü If wrong wipe and reload the APIC Enter the VLAN ID for infra network (1-4094): 3967
“acidiag touch clean” Out-of-band management configuration ...
“acidiag touch setup” Enable IPv6 for Out of Band Mgmt Interface? [N]:
Enter the IPv4 address [192.168.10.1/24]: 10.122.143.14/26
“acidiag reboot” Enter the IPv4 address of the default gateway [None]: 10.122.143.1
Enter the interface speed/duplex mode [auto]:
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Multipod Setup Verification Checklist
qVerify BGP EVPN and VPNv4 is up between pods
qVerify both unicast and multidestination interpod flows work
qVerify jumbo MTU interpod flows work
qVerify above flows work during various Spine > IPN and IPN > IPN
link failures
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Troubleshooting
Unicast Flows
Multipod Unicast Overview
Key Differences Between Single Pod Unicast
• Spines share EP’s via BGP EVPN
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Layer 2 Unicast
BD Settings - UUC Proxy, ARP Flooding Enabled, UC Routing Disabled
Pod1 Pod2
Verify first that the
flow is unicast
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Layer 2 Unicast
Ingress traffic triggers local learn
Pod1 Pod2
EP1
172.16.1.1/24 a-leaf101# show endpoint mac 0050.56a8.b003 detail | grep epg-l2-2
0050.56a8.b003 123/CiscoLive2020:vrf1 vlan-1011 0050.56a8.b003 L eth1/26 CiscoLive2020:ap1:epg-l2-2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Layer 2 Unicast
Ingress leaf updates COOP record on Spines
# ipv4.Addr
EP1 addr : 10.0.72.67/32
172.16.1.1/24 dn : topology/pod-1/node-101/sys/ipv4/inst/dom-overlay-1/if-[lo0]/addr-[10.0.72.67/32]
0050.56a8.b003 **ommitted
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Layer 2 Unicast
How does the remote pod learn about the EP?
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Layer 2 Unicast
Local spines exports to evpn
Advertised path-id 1
Path type: local 0x4000008c 0x0 ref 0, path is valid, is best path
AS-Path: NONE, path locally originated
Leaf Leaf Leaf 0.0.0.0 (metric 0) from 0.0.0.0 (192.168.1.101) Originated Locally
Origin IGP, MED not set, localpref 100, weight 32768
Received label 15761417 BD VNID
Extcommunity:
EP1 RT:5:16
172.16.1.1/24
0050.56a8.b003 Path-id 1 advertised to peers: Advertised to
192.168.2.101 192.168.2.102 Remote Pod Spines
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Layer 2 Unicast
Remote spines receive EP through EVPN
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Layer 2 Unicast
What is the Dataplane TEP/External Proxy TEP (ETEP)?
• Per-Pod anycast address a-apic1# moquery -c ipv4If -f 'ipv4.If.mode*"etep"' -x 'rsp-subtree=children'
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Layer 2 ETEP Lookup
Spine Spine COOP Forward to
Proxied Layer Lookup Points Remote Pod
2 Traffic to Remote External MAC
POD ETEP Proxy TEP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Layer 2 Unicast
Verify Remote Pod COOP Entry
a-spine3# show coop internal info repo ep | grep -B 8 -A 35 00:50:56:A8:B0:03 Spine Spine
------------------------------------------
**ommitted
Proxied L2 Traffic will forward
Pod2
EP bd vnid : 15761417
EP mac : 00:50:56:A8:B0:03 to the Pod1 External MAC-
Remote Type : MPOD proxy Address
MAC Tunnel : 10.0.0.33
IPv4 Tunnel : 10.0.0.34
IPv6 Tunnel : 10.0.0.35
ETEP Tunnel : 192.168.1.254 Leaf Leaf Leaf
**ommitted
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Layer 2 Unicast
BD Settings - UUC Proxy, ARP Flooding Enabled, UC Routing Disabled
3 Remote Spines have
2 Local Spines have COOP entry pointing to pod1-leaf101# show endpoint mac 8c60.4f02.88fc
COOP entry pointing Local Pod Leaf <no output>
to remote ETEP
Spine Spine Spine Spine
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Dynamic Tunnel Learns
Vxlan Tunnels are Created 3 Ways
id : tunnel1
dest : 10.0.72.67
Remote Pod Endpoint Learns idRequestorDn : sys/inst-overlay-1/db-dtep/dtep-[10.0.72.67]
id : tunnel1
Remote POD dest : 10.0.72.64
idRequestorDn : sys/bgp/inst/dom-overlay-1/db-dtep/dtep-[10.0.72.64]
External Routes
a-leaf205# moquery -c tunnelIf -f 'tunnel.If.id=="tunnel1"'
# tunnel.If
id : tunnel1
Local POD ISIS dest : 10.0.152.64
idRequestorDn : sys/isis/inst-default/dom-overlay-1/lvl-l1/db-dtep/dtep-[10.0.152.64]
Database
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Dynamic Tunnel Learns
Endpoint Created Tunnels
ping 172.16.1.2
Pod1 Pod2
TEP Pool: TEP Pool:
10.0.0.0/17 10.0.128.0/17
TEP: 10.0.72.67 TEP: 10.0.200.67
Leaf Leafs install white-list for remote Leaf
TEP ranges
EP1 EP1
172.16.1.1/24 172.16.1.2/24
0050.56a8.b003 8c60.4f02.88fc
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Dynamic Tunnel Learns
Endpoint (Dataplane) Created Tunnels
Outer Dst IP Outer Src IP Inner Dst IP Inner Src IP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Layer 2 Unicast
Remote Leaf Installs EP to Source
addr : 10.0.72.67/32
dn : topology/pod-1/node-101/sys/ipv4/inst/dom-overlay-1/if-[lo0]/addr-[10.0.72.67/32]
**ommitted
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Layer 2 Unicast
Return Path…
2 Spines simply
provide transit
Pod1 Pod2
3 Pod1 Leaf installs
tunnel and remote
learn to pod 2 leaf 1 Pod2 Leaf Forwards
Based on Remote Learn
EP1 EP2
172.16.1.1/24 172.16.1.2/24
0050.56a8.b003 8c60.4f02.88fc
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Using Ftriage to Troubleshoot Multipod (14.2+)
*Recommended with EX or Later Hardware
Look for bridged flow
ingressing 101 or 103
Frame seen on
a-apic1# ftriage bridge -ii LEAF:101,103 -dip 172.16.1.2 -sip 172.16.1.1 leaf101
Starting ftriage
ftriage: main:839 L2 frame Seen on a-leaf101 Ingress: Eth1/30 (Po15) Egress: Eth1/54 Vnid: 16056274
ftriage: main:242 ingress encap string vlan-1062 Frame seen on spine2
ftriage: main:839 L2 frame Seen on a-spine2 Ingress: Eth1/25 Egress: Eth1/31 Vnid: 16056274
ftriage: fib:332 a-spine2: Transit in spine
ftriage: unicast:1458 a-spine2: Infra route 10.0.200.67 present in RIB
ftriage: unicast:1681 a-spine2: Packet is exiting the fabric through {a-spine2: ['Eth1/31']}
ftriage: main:839 L2 frame Seen on a-spine3 Ingress: Eth1/29 Egress: LC-1/3 FC-22/0 Port-1 Vnid: 16056274
ftriage: fib:332 a-spine3: Transit in spine
Frame seen on pod2 spine3
ftriage: unicast:1458 a-spine3: Infra route 10.0.200.67 present in RIB
ftriage: unicast:1774 L2 frame Seen on FC of node: a-spine3….
ftriage: main:622 Found peer-node a-leaf205 and IF: Eth1/53 in candidate list
ftriage: main:839 L2 frame Seen on a-leaf205 Ingress: Eth1/53 Egress: Eth1/31 Vnid: 11371
ftriage: main:522 Computed egress encap string vlan-1039
ftriage: main:332 Egress BD(s): jy:cl1 Frame seen on pod2 leaf205
ftriage: unicast:1833 a-leaf205: Dst EP is local
ftriage: misc:657 a-leaf205: EP if(Eth1/31) same as egr if(Eth1/31) Forwards out eth1/31
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Pod 1 Verifications
Troubleshooting Scenario:
EP’s cannot communicate in L2 BD
1 Is communication unicast or multi-destination?
a-leaf101# show endpoint mac 8c60.4f02.88fc Ingress leaf has no remote learn
<no entry>
name : bd-L2-2
dn : uni/tn-CiscoLive2020/BD-bd-L2-2
unkMacUcastAct : proxy UUC set to “Proxy”
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Pod 1 Verifications
Troubleshooting Scenario:
EP’s cannot communicate in L2 BD
2 Does Local Pod Spine have the EP?
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Pod 2 Verifications
Troubleshooting Scenario:
EP’s cannot communicate in L2 BD
3 Does Remote Pod Spine have the EP?
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Troubleshooting Scenario: Pod 1 or Pod2
Verifications
EP’s cannot communicate in L2 BD
4 Is EVPN up between Pods?
Next Steps…
• Do the local spines have routes
to remote spines?
• Does IPN support jumbo MTU?
• Can spines ping between each
other?
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Layer 3 Unicast
…nearly identical to layer 2 unicast
• Differences from Layer 2
• VRF Lookup rather than BD lookup
• VRF VNID used instead of BD VNID
• Spines trigger ARP Glean if Dst is Unknown (leverages fabric multicast)
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Layer 3 Unicast – Glean Scenario
BD Settings - UC Routing Enabled
Next-hop is spine
Proxy
3a-leaf101# show isis dtep vrf overlay-1 | grep 10.0.120.34
Spine Spine
10.0.120.34 SPINE N/A PHYSICAL,PROXY-ACAST-V4
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Layer 3 Unicast – Glean Scenario No COOP Entry! This
will trigger a Glean
BD Settings - UC Routing Enabled
Local Spines have no a-spine1# show coop internal info ip-db | grep -F -B 1 -A 15 “172.16.2.2"
COOP entry for Dst IP
Spine Spine Spine Spine
Pod1 Pod2
172.16.2.2 not
Leaf Leaf Leaf Leaf Leaf Leaf learned yet
EP1 EP2
root@vm1:/home/joyo# ping 172.16.2.2 172.16.2.2/24
172.16.1.1/24
0050.56a8.b003 PING 172.16.2.2 (172.16.2.2) 56(84) bytes of data. 8c60.4f02.88fc
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Layer 3 Unicast
What is a Glean?
• If the Spines do not have an IP learn
and…
• The destination IP is within a deployed BD Subnet
ü The spine floods the proxied request with a special ethertype
ü Gleans flooded to 239.255.255.240 (*see hidden slide)
ü Leafs with destination subnet generate ARP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
ERSPAN of Spine > IPN Link
Inter-Pod Glean Custom Ethertype
for Gleans
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Layer 3 Unicast – Glean Scenario
IPN Must Route 239.255.255.240 (*see Troubleshooting Multidestination Flows Section)
IPN1# show run | grep 239
ip pim rp-address 192.168.100.1 group-list 239.0.0.0/8 bidir
ip pim rp-address 10.10.1.1 group-list 239.0.0.0/8 bidir
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Using Ftriage to Troubleshoot Multipod (14.2+)
L3 Proxy/Glean Scenario
Look for routed flow
ingressing 101 or 103 Frame seen on
leaf103
a-apic1# ftriage route -ii LEAF:101,103 -dip 172.16.2.3 -sip 172.16.1.1
ftriage: main:839 L3 packet Seen on a-leaf103 Ingress: Eth1/30 (Po15) Egress: Eth1/49 Vnid: 2588674
ftriage: main:242 ingress encap string vlan-1062
ftriage: main:301 Ingress Ctx: jy:vrf11
ftriage: main:933 SIP 172.16.1.1 DIP 172.16.2.3 Dst Unknown, Proxy!
ftriage: unicast:973 a-leaf103: <- is ingress node
Seen on spine1
ftriage: unicast:1194 a-leaf103: Dst EP is unknown - proxy
ftriage: main:839 L3 packet Seen on a-spine1 Ingress: Eth2/29 Egress: LC-2/3 FC-23/0 Port-1 Vnid: 2588674
ftriage: fib:323 a-spine1: EP not found in COOP! for VRF VNID: 2588674
ftriage: unicast:1373 a-spine1: EP is unknown in COOP. Ftriage will exit but continue with further fault isolation
ftriage: unicast:1412 a-spine1: Egress node not provided. Cannot check local EP. Exiting!
ftriage: unicast:1413 : Ftriage Completed with hunch: Check if local EP learnt on egress node(s)
EP not in COOP!
ü EP Not in COOP, gleans should be generated.
Check local learn on egress leaf
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Troubleshooting
Multidestination Flows
Multipod Multicast
What does Multipod use BUM for?
• Unknown Unicast Flooding
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
IPN Multicast Control-plane
• Spines act has multicast hosts (IGMP only)
• Spines join fabric multicast groups (Gipo’s)
• IPN’s receive Joins
• IPN’s send PIM joins to RP
• PIM Bidir is used so no (S,G)
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
What is a Gipo?
• Multicast group allocated per-VRF and per-BD.
• Used for all flooded traffic
# fv.BD
name : bd-L3-1
bcastP : 225.0.80.64
dn : uni/tn-CiscoLive2020/BD-bd-L3-1
ipLearning : yes
multiDstPktAct : bd-flood
unicastRoute : yes
unkMacUcastAct : proxy
unkMcastAct : flood
v6unkMcastAct : flood
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
IPN Multicast Control-plane
IPN
RP
Pod1 Pod2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
IPN Multicast Dataplane
IPN
All Multicast Dataplane
Goes Through RP
RP
Pod1 Pod2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
IPN Multicast Control-plane
Only one spine in each pod joins each group
Pod1 Pod2
IPN IPN IGMP Join
IGMP Join
BD Gipo Ex: BD Gipo Ex:
225.0.80.64 225.0.80.64
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
IPN Multicast Control-plane
Only one spine in each pod joins each group
Pod1 Pod2
IPN IPN IGMP Join
IGMP Join
BD Gipo Ex: BD Gipo Ex:
225.0.80.64 225.0.80.64
IPN1# show ip mroute 225.0.80.64 vrf IPN IPN1# show ip igmp groups 225.0.80.64 vrf IPN
IP Multicast Routing Table for VRF “IPN"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
(*, 225.0.80.64/32), bidir, uptime: 13:00:48, igmp ip pim Group Address Interface Uptime Expires Last Reporter
Incoming interface: loopback1, RPF nbr: 192.168.100.1 225.0.80.64 Ethernet1/1.4 13:02:14 00:04:02 192.168.1.0
Outgoing interface list: (count: 3)
Ethernet8/2, uptime: 01:34:42, pim
loopback1, uptime: 13:00:48, pim, (RPF)
Ethernet1/1.4, uptime: 13:00:48, igmp
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
IPN Multicast Control-plane
RPF for all IPN’s must point to same RP
IPN1 IPN3
Pod1 IGMP Join IGMP Join Pod2
IPN3# show ip mroute 225.0.80.64 vrf IPN IPN3# show ip pim rp 225.0.80.64 vrf IPN
IP Multicast Routing Table for VRF "IPN" PIM RP Information for group 225.0.80.64 in VRF "IPN"
RP: 192.168.100.1, (1) RPF must not
(*, 225.0.80.64/32), bidir, uptime: 01:34:35, igmp ip pim
point to ACI
Incoming interface: Ethernet8/25, RPF nbr: 10.255.0.0 IPN3# show ip route 192.168.100.1 vrf IPN
Outgoing interface list: (count: 2) 192.168.100.0/30, ubest/mbest: 1/0
Ethernet8/25, uptime: 01:34:35, pim, (RPF) *via 10.255.0.0, Eth8/25, [110/5], 13:01:42, ospf-IPN, intra
Ethernet1/17.4, uptime: 01:34:35, igmp
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
IPN Multicast Control-plane
Phantom RP
• Bidir PIM doesn’t support multiple RP’s
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
IPN Multicast Control-plane
Phantom RP RP Addr - 192.168.255.1
IPN1# show run int lo1 IPN3# show run int lo1
IPN2# show run int lo1 IPN2 IPN4 IPN4# show run int lo1
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Common Multicast Problems
Issue #1: RP Address Exists on Multiple Routers
IPN1# show run int lo1 RP Addr - 192.168.255.1
interface loopback1 IPN1 IPN1# show ip route 192.168.255.1 vrf IPN
ip address 192.168.255.1/29 IP Route Table for VRF "IPN"
ip ospf network point-to-point
ip router ospf IPN area 0.0.0.0 192.168.100.1/32, ubest/mbest: 1/0, attached
ip pim sparse-mode *via 192.168.100.1, Lo1, [0/0], 21:01:48, local
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Common Multicast Problems
Issue #2: RP Loopback not OSPF P2P Network
IPN1# show run int lo1 RP Addr - 192.168.255.1
interface loopback1 IPN1 IPN1# show ip route 192.168.255.1 vrf IPN
ip address 192.168.255.2/29 IP Route Table for VRF "IPN"
ip router ospf IPN area 0.0.0.0
ip pim sparse-mode 192.168.100.0/29, ubest/mbest: 1/0, attached
*via 192.168.100.2, Lo1, [0/0], 21:15:36, direct
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Common Multicast Problems
Issue #2: RP Loopback not OSPF P2P Network
• Loopbacks advertise /32 by default
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Common Multicast Problems
IPN3# show ip mroute 225.0.80.64 vrf IPN
Issue #3: RPF Points to ACI
(*, 225.0.80.64/32), bidir, uptime: 00:00:26, igmp ip pim
High Speed Link: Cost 1 IPN1
IPN IPN3 Incoming interface: Ethernet1/1.4, RPF nbr: 192.168.1.0
Outgoing interface list: (count: 2)
Ethernet1/1.4, uptime: 00:00:26, igmp, (RPF)
Low Speed Link: Cost 10
IPN2 RP
Spine Spine
Pod1 Pod2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Common Multicast Problems
Issue #3: RPF Points to ACI
• Spines don’t run PIM so not valid RPF
• Applicable when single spine is connected to multiple IPN’s
RPF is Eth1/1.4 (ACI) No PIM Neighbor on that Link
IPN3# show ip mroute 225.0.80.64 vrf IPN IPN1# show ip pim int eth1/1.4 brief vrf IPN
PIM Interface Status for VRF “IPN"
(*, 225.0.80.64/32), bidir, uptime: 00:00:26, igmp ip pim Interface IP Address PIM DR Address Neighbor
Incoming interface: Ethernet1/1.4, RPF nbr: 192.168.1.0 Count
Outgoing interface list: (count: 2) Ethernet1/1.4 192.168.1.1 192.168.1.1 0
Ethernet1/1.4, uptime: 00:00:26, igmp, (RPF)
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Common Multicast Problems
Issue #3: RPF Points to ACI
IPN1
IPN IPN3
High Speed Link: Cost 1
Make IPN-IPN links have
Low Speed Link: Cost 10
equal or better OSPF Cost
IPN2 RP
Spine Spine
Pod1 Pod2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Troubleshooting
External Routed
Communication
External Routed L3out Control-Plane
Almost the same as traditional L3outs
• External Routes redistributed into Fabric BGP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
External Routed L3out Control-Plane
Spines Reflect VPNv4
3 Paths between Pods
How do internal Leafs
Spine Spine Spine learn external routes?
Pod1 Pod2
Internal Leafs Import Border Leaf Exports
4 Routes from BGP into BGP and sends
2 to Spines
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
External Routed L3out Control-Plane
External Route on Internal Leaf
a-leaf101# show bgp ipv4 unicast 10.13.13.13/32 vrf CiscoLive2020:vrf1
Next-hop, this is the
Advertised path-id 1, VPN AF advertised path-id 1
Border Leaf TEP
Path type: internal adv path ref 2, path is valid, is best path
Imported from 10.0.200.67:10:13.13.13.13/32
AS-Path: NONE, path sourced internal to AS
10.0.200.67 (metric 64) from 10.0.64.64 (192.168.1.102)
Origin incomplete, MED 5, localpref 100, weight 0
Received label 0
Imported Route-target Received path-id 2 Source route AD is 110
Extcommunity: (must be OSPF)
RT:65000:2818051
Vxlan Vnid used for COST:pre-bestpath:165:2415919104 BGP Route-Reflector
traffic using this route VNID:2818051 Cluster-list, one for
COST:pre-bestpath:162:110 each pod
Originator: 10.0.200.67 Cluster list: 192.168.1.102 192.168.2.254
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
External Routed L3out Control-Plane
Tunnel Built by BGP on Internal Leaf
a-leaf101# show ip route 10.13.13.13 vrf CiscoLive2020:vrf1
IP Route Table for VRF "CiscoLive2020:vrf1"
'*' denotes best ucast next-hop
a-leaf101# vsh
a-leaf101# show bgp internal event-history objstore | grep a00c843 Dest IP in hex
Initial BGP 2019 Apr 2 21:12:30 bgp 65000 [58156]: TID 58302: (0) OBJ: bgp_dtep_add: tep=a00c843
Tunnel Creation
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
External Routed L3out Control-Plane
How do Border Leafs forward to internal Leafs?
• Exactly the same as non-multipod…
• Bridge Domain Static Route Pushed to Border Leaf by Contract
• Border Leafs Redistributes (if configured) into external protocol
• External > Internal traffic hits EP learn or BD static (proxy) route
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
External Routed L3out Control-Plane
How do Border Leafs forward to internal Leafs?
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Troubleshooting TIP
When Troubleshooting Layer 3 Flows Always…
1) Check if there is an Endpoint Learn
If not then…
2) Check if there is a BD (pervasive) static route
If not then…
3) Check if there is an External Route
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Common Multipod L3out Problems
Issue #1: Asymmetric Routing with Active/Active Pods
Spine Spine
• Both Pods advertise same
Pod1 Pod2 BD Subnet
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Common Multipod L3out Problems
Issue #1: Asymmetric Routing with Active/Active Pods
Spine Spine Implement Host Route
Pod1 Pod2 Advertisement
BD1 BD1 • Pods advertise local /32 EP
10.1.1.0/24 10.1.1.0/24 information
• Requires GOLF or Host
Leaf Leaf Leaf Leaf Border Route Feature (HBR
in 4.0)
Pod 1 Advertises
10.1.1.1/32
EP1
10.1.1.1/24
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Common Multipod L3out Problems
Issue #2: Stretched L3out VIP Failover
IPN Forwards GARP
Spine Spine
Two Common Problems Here
Pod1 Pod2 • Same encap vlan not
deployed for each vlan –
breaks flooded traffic
• IPN isn’t routing multicast
Leaf Leaf Leaf Leaf properly
New Active FW
Sends GARP
Pod 1 Leafs don’t see GARP,
still think local FW is active
Standby VIP
Active VIP Active VIP
10.2.2.2
10.2.2.2 Pod 2 Becomes Active 10.2.2.2
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Common Multipod L3out Problems
Issue #2: Stretched L3out VIP Failover
Which VNID and Gipo should the l3out use?
a-apic1# moquery -c fvIfConn -f 'fv.IfConn.dn*"uni/tn-CiscoLive2020/out-EIGRP/"' “EIGRP” is the name
Total Objects shown: 2 of the L3out
# fv.IfConn
bcastP : 225.1.188.208
dn : uni/epp/rtd-[uni/tn-CiscoLive2020/out-EIGRP/instP-defaultNet]/node-101/stpathatt-[shared-
5596-A-VPC]/conndef/conn-[vlan-1052]-[52.52.52.101/24]
extEncap : vxlan-15466402
# fv.IfConn
bcastP : 225.1.188.208
dn : uni/epp/rtd-[uni/tn-CiscoLive2020/out-EIGRP/instP-defaultNet]/node-205/stpathatt-[shared-
5596-A-VPC]/conndef/conn-[vlan-1052]-[52.52.52.103/24]
extEncap : vxlan-15466402
The same VNID and GIPO is extended to nodes 101 and 205
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Common Multipod L3out Problems
Issue #2: Stretched L3out VIP Failover
If there’s a problem check these things…
• Ensure an SVI is used for the l3out (no flooding for routed interfaces)
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
Quality of Service
ACI QoS Overview
Key Points
• Fabric QoS is based on COS and DEI bits in outer L2 header
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
ACI QoS Overview
Inner Header iVXLAN Outer Header Fabric QOS
flags Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID DIP SIP 802.1Q SMAC DMAC
EPG UDP
Dot1p Preserve Global Access Policies Causes egress leaf to rewrite cos
to original value when forwarding
QoS Class App EPG, Contract, Subject Defines prioritization of traffic
through the fabric
Custom QoS App EPG Re-marks traffic based on
incoming COS or DSCP
Target DSCP (L3out) L3out, Contract, Subject Sets the DSCP value
DSCP Class-Cos Infra > Networking > Protocols Spines re-map QoS of traffic
Translation Policy going to and coming from IPN/ISN
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
ACI QoS – Preserve COS
Egress leaf
rewrites COS
based on DSCP
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
ACI Forwarding and QoS – Preserve COS
Layer 2 COS encoded into most significant 3 bits of DSCP
flags
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC VNID DSCP DIP SIP 802.1Q SMAC DMAC
EPG
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Pre-4.0 COS 6 Problem Fix? Configure “DSCP class-cos
translation policy for L3 traffic”
Last hop IPN router The spine will map the outer COS
writes COS based on value to a new DSCP class on
egress and map DSCP to COS in
DSCP
Datacenter interconnect ingress
…DSCP 48 = COS6 4
(IPN, ISN)
DC1 treats 3
IP packet
packet as with DSCP 48
iTraceroute
5
Data Data
Center 1 Center 2
2
Leaf forwards frame
towards DC1 with
1 COS 0 and an outer
Frame with DSCP of 48
COS 6 set 0b110 000
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
DSCP – COS
Translation Policy
ü COS 6 Problem
solved by using
DSCP – COS
Translation Policy
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
After 4.0 Software…
• All devices trust DSCP Pre-4.0
markings set on ingress leaf Traceroute,
Datacenter Spine
• QoS class is derived from interconnect COS6 + not forwarded
DSCP 48 on egress leaf
DSCP (IPN, ISN)
due to COS6
• Spine rewrites COS
received from IPN based on After 4.0
DSCP
Datacenter Spine
• Traceroute is DSCP 6 so COS6 + Whichever
interconnect class DSCP 48
COS 6 + DSCP 48 is DSCP 48
(IPN, ISN) maps to
forwarded normally
ü DSCP – COS
Translation Policy
Not Required
#CiscoLive DGTL-BRKACI-2934 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Thank you
#CiscoLive
#CiscoLive