Professional Documents
Culture Documents
03 - VXLAN Part III The Underlay Network - Multidestination Traffic Anycast-RP With PIM
03 - VXLAN Part III The Underlay Network - Multidestination Traffic Anycast-RP With PIM
03 - VXLAN Part III The Underlay Network - Multidestination Traffic Anycast-RP With PIM
RP with PIM
The role of the Underlay Network, related to BUM traffic in the VXLAN fabric, is to
transport ARP, ND, DHCP and other Layer 2 BUM (Broadcast, Unknown Unicast, and
Multicast) traffic between the hosts connected to different VTEPs. For the Layer 3
Multicast traffic between hosts, there should be an overlay Multicast routing design. This
chapter shows how an Anycast-RP with PIM can be used in a VXLAN fabric. In figure 1,
we can see our example topology used in this chapter. There are two Spine switches,
which shares the same Anycast-RP IP address and belongs to the same “Anycast-RP set”
group (Loopback 238). In addition to that, there is an another loopback interface, which
must be unique in each Spine (Loopback 511 and 512). These addresses are used as an
Anycast-RP group member Id. Both addresses, shared and unique, needs to be
reachable for all switches. Complete configuration can be found from the Appendix 1 at
the end of the document.
I am going to build an Underlay network Multicast routing using Anycast-RP with PIM.
During the implementation process, I am also going to explain the theory part.
Figure 2: Anycast-RP ip addresses.
feature pim
Step-2: Configure a loopback interface for an Anycast-RP shared between the cluster
member. This configuration is identical in both Spine switches. Since this interface is
used as an RP address, it has to be reachable for all switches. We enable both PIM-SM
and OSPF on the new Loopback interface.
!
interface loopback238
description ** Anycast-RP address **
ip address 192.168.238.238/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
Step-3: Configure Unique IP address for each Anycast-RP cluster member and enable
PIM-SM and OSPF on it. This address is used as a cluster member ID. Also, define the
other Anycast-RP cluster members. Our Example configuration is taken from Spine-11.
Step-4: Configure the IP address of the RP on all switches and optionally define the
Multicast groups for this RP. Also, make sure that PIM-SM and OSPF are enabled on each
loopback interface and on each inter-switch link shown in figure 4.
interface nve1
no shutdown
source-interface loopback100
member vni 10000
mcast-group 238.0.0.10
!
vlan 10
vn-segment 10000
Now we are ready to see how this really works. What I am going to do is shut down the NVE
interfaces on both VTEPs and turn on the packet capture on all inter-switch links. Then I am going
to re-open the NVE interfaces, first in VTEP-101 and then in VTEP-102. This way we can see the
Multicast Join > Register > Register-Stop process of VTEPs towards the Anycast-RP.
At this moment NVE interfaces are down. Here is the MRIB from both Spine switches.
Spine-11# sh ip mroute
IP Multicast Routing Table for VRF "default"
Spine-12# sh ip mroute
IP Multicast Routing Table for VRF "default"
Now I am bringing up the NVE 1 interface on VTEP1. When the NVE 1 is up, VTEP starts to join
process to group 238.0.0.10. The PIM Join message (to group 238.0.0.10) is sent only towards
Spine-11 based on 5-tuple hash (Figure 6 and Capture 1). The source address is VTEP-101s
Underlay Network IP address 192.168.0.101 and the destination address is 224.0.0.13 (All PIM
routers) This way the VTEP-101 joins the RPT (Root Path Tree).
Figure 6: PIM Join from VTEP-101.
Now the VTEP has been joined to RPT of group 238.0.0.10. Then it sends a PIM registration
message, this time VTEP-101 chooses another link towards Spine-12. Why is that? In figure 7 and
capture 2 we can see that PIM registration packet is sent as a unicast towards the Anycast-RP
address 192.168.238.238 (not to 224.0.0.13) by using NVE 1 ip address as a source. This way the
5-tuple hash is different and may end up to different link (as in case of our example).
Figure 7: PIM Register message from VTEP-101 to Spine-12
Next, the Spine-12 will instruct VTEP-101 to stop encapsulation the Multicast traffic to group by
Sending Register-Stop message. As can be seen from figure 8 and capture 3, it uses Anycast-RP
address as a source and VTEP-101 NVE as a destination as can be seen from Figure 8 and Capture
3. This is the reason why all previously configured Loopback addresses have to be reachable for all
switches.
Figure 8: PIM Register –Stop from Spine-12 to VTEP-101
From VTEP point of view, we are done. But how Spine-11 knows that VTEP-101 is registered to
group 238.0.0.10 since the registration is sent only to Spine-12? The answer can be found from
the RFC4610 (section 3). What happens when Spine-12 receives the PIM Register message from
VTEP-101? It will forward it to its Anycast-RP cluster member Spine-11. This can be seen from
figure 9 and captures 4 and 5. It uses its Anycast-RP Unique address 192.168.238.12 as a source
and the destination address is Spine-11 Anycast-RP unique address 192.168.238.11. Receiving
end (Spine-11) can verify that message is received from valid Anycast-RP cluster peer from
messages source address (we have configured peers statically). This also explains why the Unique
address has to be known by every switch.
Figure 9: VTEP-101 PIM register message relayed by Spine-12 to Spine-11.
Capture 4: VTEP-101 PIM register message relayed by Spine-12 to Spine-11 (Part 1).
Capture 5: VTEP-101 PIM register message relayed by Spine-12 to Spine-11 (Part 2).
And now the PIM registration process is ready. We can see that VTEP-101 is known as a source for
Group 238.0.0.10
Spine-11# sh ip mroute
IP Multicast Routing Table for VRF "default"
Spine-12# sh ip mroute
IP Multicast Routing Table for VRF "default"
When enabling Interface NVE 1 on VTEP-102, the same Join > Register > Register-Stop process is
done and both VTEPs has joined to group 238.0.0.10. In this phase both VTEPs are known as a
source for the group 238.0.0.10.
Spine-11#
Spine-11# sh ip mroute
IP Multicast Routing Table for VRF "default"
Spine-12# sh ip mroute
IP Multicast Routing Table for VRF "default"
Even though not shown in previous pictures, there is a Host-1 with IP 192.168.11.11/24 connected
to interface eth 1/3 on VTEP-101. In VTEP-102 there is a Host-2 with IP 192.168.11.12/24
connected to Interface eth1/3. For the verification that our L2VNI is up and also capable of
transport L2 BUM (ARP in this case), we ping from Host-1 to Host-2. As can be seen, we lost a
couple of ping packets at the beginning because of the ARP process, but then it starts to work.
Host-1#ping 192.168.11.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.11.12, timeout is 2 seconds:
...!!
Success rate is 40 percent (2/5), round-trip min/avg/max = 18/22/26 ms
Host-1#
And just for the verification, here are couple captures, where we can see that ARP messages are
encapsulated with VXLAN header by VTEP 101 and those are sent to Group 238.0.0.10 with a
source address of interface NVE 1.
Capture 6: ARP request sent by Host 192.168.11.11 encapsulated by VTEP-101.
Here we can see the response from VTEP-102. ARP reply is sent as a Unicast message to VTEP-
101 NVE interface address. Do not get confused even though the frame numbers are same (Frame
8) in both captures. The ARP request message was sent over the different link than where the ARP
Reply message was received (ECMP).
Capture 7: ARP reply from Host 192.168.11.12 to Host 192.168.11.11 sent by VTEP-102.
I will describe the Flood & Learn process in my becoming posts. But before that, I will write articles
about PIM BiDir and Ingress Replication.
References:
RFC 4610: Anycast-RP Using Protocol Independent Multicast (PIM)
Building Data Center with VXLAN BGP EVPN – A Cisco NX-OS Perspective
ISBN-10: 1-58714-467-0
Appendix 1. Configurations
Leaf-101
Leaf-101# sh run
version 7.0(3)I7(1)
hostname Leaf-101
vdc Leaf-101 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
no password strength-check
username admin password 5
$5$4dqvZbsf$hSqYx5Vb6kNO/UFBzuK2CfAVzDYW7iJMisF3GboHwn
4 role network-admin
ip domain-lookup
ip host Leaf-101 192.168.0.101
ip host Leaf-102 192.168.0.102
ip host Spine-11 192.168.0.11
ip host Spine-12 192.168.0.12
snmp-server user admin network-admin auth md5
0x223cfb63ca87c5b4856c960235329cff
priv 0x223cfb63ca87c5b4856c960235329cff localizedkey
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
interface nve1
no shutdown
source-interface loopback100
member vni 10000
mcast-group 238.0.0.10
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
switchport access vlan 10
!
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.101/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback100
description ** VTEP/Overlay **
ip address 192.168.100.101/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.101
name-lookup
Leaf-101#
Leaf-102
Leaf-102# sh run
version 7.0(3)I7(1)
hostname Leaf-102
vdc Leaf-102 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
interface nve1
no shutdown
source-interface loopback100
member vni 10000
mcast-group 238.0.0.10
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/3
switchport access vlan 10
!
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.102/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback100
description ** VTEP/Overlay **
ip address 192.168.100.102/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.102
name-lookup
Leaf-102#
Spine-11
Spine-11# sh run
version 7.0(3)I7(1)
hostname Spine-11
vdc Spine-11 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature ospf
feature pim
no password strength-check
username admin password 5
$5$60DVUPIV$uZWPu6ufHQOJSG18SK5b9/5kpZnV5E4/EFapzQP5CI
/ role network-admin
ip domain-lookup
ip host Leaf-101 192.168.0.101
ip host Spine-12 192.168.0.12
ip host Leaf-102 192.168.0.102
snmp-server user admin network-admin auth md5
0xd177fd3448eab21dd2feb16d54938469
priv 0xd177fd3448eab21dd2feb16d54938469 localizedkey
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
!
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.11/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback238
description ** Anycast-RP address **
ip address 192.168.238.238/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback511
description ** Unique Address for Anycast-RP **
ip address 192.168.238.11/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.11
name-lookup
Spine-11#
Spine-12
Spine-12# sh run
version 7.0(3)I7(1)
hostname Spine-12
vdc Spine-12 id 1
limit-resource vlan minimum 16 maximum 4094
limit-resource vrf minimum 2 maximum 4096
limit-resource port-channel minimum 0 maximum 511
limit-resource u4route-mem minimum 128 maximum 128
limit-resource u6route-mem minimum 96 maximum 96
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8
feature ospf
feature pim
no password strength-check
username admin password 5
$5$CnfXhejK$UE7azuRSVXBSEVTPYeW4fI1.UTH3x69GU22CBnVhOA
8 role network-admin
ip domain-lookup
ip host Leaf-101 192.168.0.101
ip host Spine-12 192.168.0.12
ip host Spine-11 192.168.0.11
ip host Leaf-102 192.168.0.102
snmp-server user admin network-admin auth md5
0x40c5b687ff82eb6f487bbafc8a2cf722
priv 0x40c5b687ff82eb6f487bbafc8a2cf722 localizedkey
rmon event 1 description FATAL(1) owner PMON@FATAL
rmon event 2 description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 description ERROR(3) owner PMON@ERROR
rmon event 4 description WARNING(4) owner PMON@WARNING
rmon event 5 description INFORMATION(5) owner PMON@INFO
interface Ethernet1/1
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet1/2
no switchport
medium p2p
ip unnumbered loopback0
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
!
interface mgmt0
vrf member management
interface loopback0
description ** RID/Underlay **
ip address 192.168.0.12/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback238
description ** Anycast-RP address **
ip address 192.168.238.238/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback512
description ** Unique Address for Anycast-RP **
ip address 192.168.238.12/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
line console
line vty
router ospf UNDERLAY-NET
router-id 192.168.0.12
name-lookup
Spine-12#