Internal Draft

Date : 12 May 2023

Interconnecting Multiple DC sites

with Dual Fabric VxLAN Technology

Project Name : DHS Data Centres Interconnection

Project Status: Initial Phase - Providing Technical Solutions for Interconnecting
DHS data centres
Version : 1.1
Author : Rahul Siddhanak
Position : Network Design Architect

Document History
Version Date Description
Ver 1.0 24 April 2023 Interconnecting Multiple DC sites with VxLAN
Ver 1.1 12 May 2023 Configuration of Multihoming with vPC with VxLAN Multisite
Abstract

This document discusses solutions for inter-connection of multiple DHS DC sites with dual fabric VxLAN
technology. Two tasks are discussed in this document.

1. In the first task, two BGW Switches are connected to ACI Leaf Switches in both ACI DCs with Dual
Fabric vPC over VxLAN with vPC anycast GW.

2. In the second task for VxLAN Multisite Configuration, BGW Switches are connected to other site BGW
for the extension of L2/L3 network between multiple dc sites.

3. All Interconnection/Multisite configuration will be in BGW Switches only.

Task 1 : Build vPC over dual fabric VxLAN

- Two Border Gateway Switches (BGWs) which supports VxLAN technology will be dual connected to
ACI leaf switches with vPC Anycast VTEP Address on each ACI DC sites.

- Two Border Gateway Switches (BGWs) which supports VxLAN technology will be dual connected to
N7K switches with vPC Anycast VTEP Address.
Task 2 : VxLAN Multisite Configuration
Step 1 : Configure Peer-keepalive link

Configure Peer-keepalive link on BGW1

vrf context VPC-Peer-Keepalive

!
interface Ethernet1/7
no switchport
vrf member VPC-Peer-Keepalive
ip address 10.10.10.11/24
no shutdown
!
vpc domain 99
peer-keepalive destination 10.10.10.12 source 10.10.10.11 vrf VPC-Peer-Keepalive

Configure Peer-keepalive link on BGW2

vrf context VPC-Peer-Keepalive

!
interface Ethernet1/7
no switchport
vrf member VPC-Peer-Keepalive
ip address 10.10.10.12/24
no shutdown
!
vpc domain 99
peer-keepalive destination 10.10.10.11 source 10.10.10.12 vrf VPC-Peer-Keepalive

Step 2 : Configure vPC Peer-Link

Configure vPC Peer-Link on BGW1

interface port-channel80
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet1/11
description ** Po80 member - vPC PEER-link **
switchport mode trunk
channel-group 80 mode active
!
interface Ethernet1/12
description ** Po80 member - vPC PEER-link **
switchport mode trunk
channel-group 80 mode active

Configure vPC Peer-Link on BGW2

interface port-channel80
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet1/11
description ** Po80 member - vPC PEER-link **
switchport mode trunk
channel-group 80 mode active
!
interface Ethernet1/12
description ** Po80 member - vPC PEER-link **
switchport mode trunk
channel-group 80 mode active

Step 3 : Configuration of Multihoming with vPC

*Configuration on BGW1*

feature vpc
feature lacp

interface port-channel100
switchport mode trunk
vpc 100
!
interface Ethernet1/5
description ** Link to ACI Leaf 1**
switchport mode trunk
channel-group 100

interface Ethernet1/6
description ** Link to ACI Leaf 2**
switchport mode trunk
channel-group 100

*Configuration on BGW2*

feature vpc
feature lacp

interface port-channel200
switchport mode trunk
vpc 200
!
interface Ethernet1/5
description ** Link to ACI Leaf 1**
switchport mode trunk
channel-group 200

interface Ethernet1/6
description ** Link to ACI Leaf 2**
switchport mode trunk
channel-group 200
Step 4 : VTEP redundancy with vPC

*Configuration on BGW1*

router bgp XXXXX

address-family l2vpn evpn
advertise-pip
!
interface nve1
advertise virtual-rmac

interface loopback100
description ** VTEP/Overlay **
ip address 192.168.100.101/32
ip address 192.168.100.100/32 secondary
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode

*Configuration on BGW2*

router bgp XXXXX

address-family l2vpn evpn
advertise-pip
!
interface nve1
advertise virtual-rmac

interface loopback200
description ** VTEP/Overlay **
ip address 192.168.100.102/32
ip address 192.168.100.100/32 secondary
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode

Step 5 : Define loopback ip address as the Source ip address of each DC Site using loopback command on BGW
switches
- Loopback1 at DC1, Loopback2 at MLK and Loopback3 at LACUSE

Step 6 : Define Site ID for each DC site

DC1 MLK LACUSE
evpn multisite border-gateway 1 evpn multisite border-gateway 2 evpn multisite border-gateway 3
delay-restore time 30 delay-restore time 30 delay-restore time 30
Step 7: Define eBGP peering between BGW with Site-specific Autonomous System
DC1 MLK LACUSC
router bgp XXX1 router bgp XXX2 router bgp XXX3
neighbor 2.2.2.2 neighbor 1.1.1.1 neighbor 2.2.2.2
neighbor 3.3.3.3 neighbor 3.3.3.3 neighbor 1.1.1.1
remote-as XXX2 remote-as XXX1 remote-as XXX2
remote-as XXX3 remote-as XXX3 remote-as XXX1
update-source loopback1 update-source loopback2 update-source loopback3
ebgp-multihop 5 ebgp-multihop 5 ebgp-multihop 5
Peer type fabric-extended Peer type fabric-extended Peer type fabric-extended
address-family l2vpn evpn address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended send-community extended
rewrite-evpn-rt-asn rewrite-evpn-rt-asn rewrite-evpn-rt-asn

Step 8 : Define Fabric link and Site-external DCI link

DC1 MLK LACUSC

Interface Eth 1/11 Interface Eth 1/21 Interface Eth 1/31
evpn multisite fabric-tracking evpn multisite fabric-tracking evpn multisite fabric-tracking

Interface Eth 1/12 Interface Eth 1/22 Interface Eth 1/32

evpn multisite dci-tracking evpn multisite dci-tracking evpn multisite dci-tracking

Step 9 : Use underlay IGP to advertise the BGW IP internal to each other
- Need to do configuration changes in APIC and N7K switches

Step 10 : Make sure of MTU size – 9150 on DCI link

Step 11 : All extended L2 and L3 VNI must be configured in BGW switches

DC1 MLK LACUSC
Interface nve1 Interface nve1 Interface nve1
multisite border-gateway multisite border-gateway multisite border-gateway
interface loopback1 interface loopback2 interface loopback3
member vni 111111 member vni 111111 member vni 111111
Suppress-arp Suppress-arp Suppress-arp
multisite ingress-replication multisite ingress-replication multisite ingress-replication
member vni 222222 member vni 222222 member vni 222222
suppress-arp suppress-arp suppress-arp
multisite ingress-replication multisite ingress-replication multisite ingress-replication