Professional Documents
Culture Documents
Troubleshooting Duplicate IP/Mac in MP-BGP Evpn Vxlan On Nexus9000
Troubleshooting Duplicate IP/Mac in MP-BGP Evpn Vxlan On Nexus9000
duplicate IP/Mac in
MP-BGP EVPN VxLan
on Nexus9000
Gunjan Verma
CTHDCN-2304
#CLUS
Who are we
We are Technical Consulting Engineers from the Customer
Experience (CX) Support Services organization, also known
as TAC
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Problem Definition & Symptoms
• Host Mobility –Extended Community Attribute
• MAC/IP Duplication Issue Addressed
• Conclusion
• Q&A
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Problem Definition & Symptoms
Host A can not Reach to Host C in VxLAN BGP EVPN Fabric
2019 Apr 23 23:19:45 Leaf3 l2rib: Detected duplicate host 0010.3900.7901, topology 11, during Local update, with host located at remote VTEP
10.1.255.4, VNI/EVI 10011
Spine1 Spine2
L3 Underlay
Unicast / Multicast Routing
Host-A Host-B
10.101.11.79 11.0.0.2 Host-C
0010.3900.7901 10.101.11.79
0010.3900.7901
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Host Mobility –Extended Community Attribute
neighbor a.b.c.d
remote-as 65536
NLRI:
Host MAC1,IPA update-source
NVE-IP1 loopback0
VNI 10000 SPINE-1 SPINE-2 SPINE-3 SPINE-4
EXT. Community:
ebgp-multihop 2
Encapsulation : address-family l2vpn
VxLAN evpn
Sequence 0
allowas-in 3
VTE-IP1 VTE-IP2 VTE-IP3 VTE-IP4 send-community
VTEP-2 VTEP-3 VTEP-4
VTEP-1
extended
SVIs SVIs
SVIs SVIs
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Host Mobility –Extended Community Attribute
NLRI:
Host MAC1,IPA
NVE-IP3
SPINE-2 SPINE-3 VNI 10000
SPINE-1 SPINE-4
EXT. Community:
Encapsulation : VxLAN
Sequence 1
SVIs SVIs
SVIs SVIs
MAC IP VNI NXT Encap Seq
HOP
HOST-IPA/MAC1 MAC1 IPA 10000 IP3 VxLan 1
VLAN 10 VNI 10000
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
MAC/IP Duplication Issue Addressed
• Possible two (or more) hosts being misconfigured with the same (duplicate) MAC/IP address
• Traffic originating from these hosts would trigger continuous MAC/IP moves between VTEP’s
• Result of this continuous increasing of Sequence number to infinity because of the MAC
Mobility extended community attribute
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
MAC/IP Duplication Issue Addressed
N9K-L3# show clock ; show l2route evpn mac all detail | i Flags:|Topology|0010.3900.7901
08:33:55.353 PM EST Wed April 16 2019
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link When a host is initially learned in
the VXLAN fabric the BGP EVPN
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending sequence bit is set to zero (0)
Topology Mac Address Prod Next Hop (s) PeerID Seq Number Flags NFN Bitmap
11 0010.3900.7901 Local Eth1/5 0 0 L 32
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
MAC/IP Duplication Issue Addressed-cont
N9K-L3# show clock ; show l2route evpn mac all detail | i Flags:|Topology|0010.3900.7901
08:34:46.587 PM EST Wed April 16 2019
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link When a same host is remotely
learned in the VXLAN fabric the
BGP EVPN sequence bit is set to
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending One (1)
Topology Mac Address Prod Next Hop (s) PeerID Seq Number Flags NFN Bitmap
11 0010.3900.7901 BGP 10.1.255.4 1 1 Rcv 1
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
MAC/IP Duplication Issue Addressed-cont
N9K-L3# show clock ; show l2route evpn mac all detail | i Flags:|Topology|0010.3900.7901
08:34:46.587 PM EST Wed April 16 2019
Topology Mac Address Prod Next Hop (s) PeerID Seq Number Flags NFN Bitmap
11 0010.3900.7901 BGP 10.1.255.4 1 5 Rcv 1
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
MAC/IP Duplication Issue Addressed-cont
N9K-L3# show clock ; show l2route evpn mac all detail | i Flags:|Topology|0010.3900.7901
08:36:24.002 PM EST Wed April 16 2019
Flags:(Rtr)=Router MAC; (Stt)=Static; (L)=Local; (R)=Remote; (V)=vPC link; (Dup)=Duplicate; (Spl)=Split; (Rcv)=Recv; (D)=Del Pending;
(S)=Stale; (C)=Clear
Topology Mac Address Prod Next Hop (s) PeerID Seq Number SOO Flags NFN Bitmap
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
MAC/IP Duplication Issue Addressed-cont
N9K-L3# N9K-L3# show system internal l2rib event mac | grep unfreeze Syslog of the 3 x 30 sec freezes:
[04/16/19 20:36:21.491 EST 1adbd49 7921] (11,0010.3900.7901):Added to dup list, Unfreeze timeout: 30 secs, unfreeze cycles count: 3,
currently at cycle: 1, list length 0
[04/16/19 20:36:21.491 EST adbd50 7921] (11,0010.3900.7901):Removed from dup list, Unfreeze timeout: 30 secs, unfreeze cycles
count: 3, currently at cyc e: 1, list , length 0
[04/16/19 20:39:02.522 EST 1adca9c 7921] (11,0010.3900.7901):Added to dup list, Unfreeze timeout: 30 secs, unfreeze cycles count: 3,
currently at cycle: 2, list length 1
[04/16/19 20:39:38.382 EST 1adcec5 7921] (11,0010.3900.7901):Removed from dup list, Unfreeze timeout: 30 secs, unfreeze cycles
count: 3, currently at cycle: 2, list length 0
[04/16/19 20:42:52.578 EST 1add04e 7921] (11,0010.3900.7901):Added to dup list, Unfreeze timeout: 30 secs, unfreeze cycles count: 3,
currently at cycle: 3, list length 1
[04/16/19 20:43:27.953 EST 1add055 7921] (11,0010.3900.7901):Removed from dup list, Unfreeze timeout: 30 secs, unfreeze cycles
count: 3, currently at cycle: 3, list length 0
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
MAC/IP Duplication Issue Addressed-cont
N9K-L3# show clock ; show l2route evpn mac all detail | i Flags:|Topology|0010.3900.7901
08:36:24.002 PM EST Wed April 16 2019
With this continuing permanent
Flags:(Rtr)=Router MAC; (Stt)=Static; (L)=Local; (R)=Remote; (V)=vPC link; (Dup)=Duplicate; (Spl)=Split; (Rcv)=Recv; (D)=Del Pending;
Freeze occurs
(S)=Stale; (C)=Clear
Topology Mac Address Prod Next Hop (s) PeerID Seq Number SOO Flags NFN Bitmap
2019 April 16 14:04:22.788 N9K-L3 ER-2-SYSTEM_MSG: Detected duplicate host 0010.3900.7901, topology 11, during Local
update, with host located at remote VTEP 10.1.255.4, VNI 10011 - l2rib
2019 Apr 16 14:04:08 N9K-L3 l2rib: Unfreeze limit (3) hit, MAC 24e9.b39e.a1fc in topo: 11 is permanently frozen l2rib: Unfreeze
limit (3) hit, MAC 24e9.b39e.a1fc in topo: 11 is permanently frozen
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Problem Definition & Symptoms
Host A cannot Reach to Host C same vlan in VxLAN BGP EVPN Fabric
Spine1 Spine2
L3 Underlay
Unicast / Multicast Routing
Host-A Host-B
11.0.0.99 11.0.0.2 Host-C
003a.7d4d.fe87 11.0.0.99
24e9.b39e.a1fc
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
ARP Table for Leafs
Leaf1#
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface Flags
11.0.0.99 00:01:25 003a.7d4d.fe87 Vlan11
Leaf3/Leaf4#
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface Flags
11.0.0.99 00:07:48 24e9.b39e.a1fc Vlan11
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
L2route EVPN for Local and Remote Mac
Leaf1#
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
RIB-Route in Tenant Vrf for Leafs
Leaf1#
Leaf3/Leaf4 #
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
BGP L2vpn route for Leaf1
Show bgp l2vpn evpn 11.0.0.99
BGP routing table entry for [2]:[0]:[0]:[48]:[24e9.b39e.a1fc]:[32]:[11.0.0.99]/272, version 1375
Paths: (2 available, best #1)
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW Route learned in BGP
Advertised path-id 1 table with NVE ip
Path type: external, path is valid, is best path, no labeled nexthop, in rib address of Leaf3 as
Imported from 4.4.4.4:32778:[2]:[0]:[0]:[48]:[24e9.b39e.a1fc]:[32]: Next-hop reflected by
[11.0.0.99]/272 Spine.
AS-Path: 65536 65551 , path sourced external to AS
192.168.100.100 (metric 0) from 10.10.10.10 (10.10.10.10)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10011 10099
Extcommunity: RT:23456:10011 RT:23456:10099 SOO:192.168.100.100:0 ENCAP:8
MAC Mobility Sequence:00:105 Router MAC:cc46.d621.b21f
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
BGP L2vpn route for Leaf3/Leaf4
Show bgp l2vpn evpn 11.0.0.99
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 3.3.3.3:32778 (L2VNI 10011)
BGP routing table entry for [2]:[0]:[0]:[48]:[003a.7d4d.fe87]:[32]:[11.0.0.99]/272, version 47 Route learned in
Paths: (1 available, best #1) BGP table with
Flags: (0x000212) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW NVE ip address
Advertised path-id 1 of Leaf1 as
Path type: external, path is valid, is best path, remote nh not installed, no labeled nexthop, in rib Next-hop
Imported from 30.30.30.30:32778:[2]:[0]:[0]:[48]:[003a.7d4d.fe87]:[32]:[11.0.0.99]/272 reflected by
AS-Path: 65536 65551 , path sourced external to AS
Spine.
192.168.99.99 (metric 0) from 10.10.10.10 (10.10.10.10)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10011 10099
Extcommunity: RT:23456:10011 RT:23456:10099 SOO:192.168.99.99:0 ENCAP:8
MAC Mobility Sequence:00:104 Router MAC:58f3.9ca9.1dad
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Fabric Forwarding Local HOST DB
Leaf 1#
Ip address
show fabric forwarding ip local-host-db vrf EVPN_A 11.0.0.99/32
11.0.0.99
with local
HMM routing table information for VRF EVPN_A, address family IPv4
host mac is
HMM routing table entry for 11.0.0.99/32
learned
Hosts: (1 available)
locally and
installed in
Host type: Local(Flags: 0x1420201), in Rib
RIB as Hmm
mac: 003a.7d4d.fe87, svi: Vlan11, bd: 11, phy_intf: port-channel10
in vlan 11
Leaf 3/Leaf4#
Ip address
Show fabric forwarding ip local-host-db vrf EVPN_A 11.0.0.99/32
11.0.0.99
with local
HMM routing table information for VRF EVPN_A, address family IPv4
host mac is
HMM routing table entry for 11.0.0.99/32
learned
Hosts: (1 available)
locally and
installed in
Host type: Local(Flags: 0x420201), in Rib
RIB as Hmm
mac: 24e9.b39e.a1fc, svi: Vlan11, bd: 11, phy_intf: port-channel10
in vlan 11
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Fabric Local HOST DB with Duplicate Status Set
Leaf 1#
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Fabric Local HOST DB Events
Leaf1:
2019 May 7 13:40:08.391571 hmm [29853]: [29953]: (EVPN_A) [IPv4] Received AM notification for Host 11.0.0.
99/32, mac 003a.7d4d.fe87, svi Vlan11, l2_port port-channel10, flags 0x00000000
2019 May 6 01:31:35.634444 hmm [29853]: [29953]: (EVPN_A) [IPv4] Received AM notification for Host 11.0.0.
99/32, mac 0000.0000.0000, svi Vlan11, l2_port -, flags 0x00000000
2019 May 6 00:55:40.144703 hmm [29853]: [29953]: (EVPN_A) [IPv4] Received AM notification for Host 11.0.0.
99/32, mac 003a.7d4d.fe87, svi Vlan11, l2_port port-channel10, flags 0x00000000
2019 May 6 00:53:34.054691 hmm [29853]: [29953]: (EVPN_A) [IPv4] Received AM notification for Host 11.0.0.
99/32, mac 0000.0000.0000, svi Vlan11, l2_port -, flags 0x00000000
2019 May 6 00:46:04.931445 hmm [29853]: [29953]: (EVPN_A) [IPv4] Received AM notification for Host 11.0.0.
99/32, mac 003a.7d4d.fe87, svi Vlan11, l2_port port-channel10, flags 0x00000000
Leaf3/Leaf4#
2019 May 7 16:47:18.512139 hmm [366]: [508]: (EVPN_A) [IPv4] Received AM notification for Host
11.0.0.99/32, mac 24e9.b39e.a1fc, svi Vlan11, l2_port port-channel10, flags 0x00000000
2019 May 7 16:47:17.698819 hmm [366]: [508]: (EVPN_A) [IPv4] Received AM notification for Host
11.0.0.99/32, mac 0000.0000.0000, svi Vlan11, l2_port -, flags 0x00000000
2019 May 7 16:45:09.786129 hmm [366]: [508]: (EVPN_A) [IPv4] Received AM notification for Host
11.0.0.99/32, mac 24e9.b39e.a1fc, svi Vlan11, l2_port port-channel10, flags 0x00000000
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Clear Duplicate and Frozen Entry for HOST
Command:
Clear ip arp <ip address> vrf <tenant-vrf> force-delete is required.
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Continue your education
Demos in the
Walk-in labs
Cisco campus
#CLUS CTHDECN-2304 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Thank you
#CLUS
#CLUS