ACI and GOLF Integration

Configuration Steps
Max Ardica – Principal Engineer (INSBU)
ACI and GOLF Integration
Reference Topology
AS 3

Pod1-GOLF1 Pod1-GOLF1
(135.135.135.135) (136.136.136.136)
1/2 1/1 1/2 1/1

MP-EBGP 192.168.1.0
1/11 1/12 1/11 1/12

Pod1-Spine1 Pod1-Spine2
(192.168.1.101) (192.168.1.102)

AS 65501

2
ACI and GOLF Integration
Summary of Configuration Steps

• The following steps are required to ensure a successful ACI and GOLF
integration
1. Properly configure the GOLF devices
One time Day-0
2. Create Fabric Access Policies
configuration
3. Create the L3Out in ‘infra’ Tenant
4. Create the L3Out in ‘Overlay’ Tenant
Recurring configuration
5. ‘Overlay’ Tenant VRF Configuration
6. Verification of Config Pushed on GOLF Device

3
GOLF Device Configuration
Nexus 7000

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 1a
GOLF Device Configuration (Nexus 7000)
Basic Config

install feature-set mpls
install feature-set fabric
feature-set mpls
feature-set fabric
feature fabric forwarding
nv overlay evpn
feature bgp
feature interface-vlan
feature nv overlay
feature vni
feature ospf
feature ipp
feature mpls l3vpn
feature mpls ldp
system bridge-domain 100-3000
system fabric bridge-domain 2000-3000
  BDs to VNIs
Enable # fabric facing interface Setup infra connectivity
interface e1/2.4
no shutdown
encapsulation dot1q 4
ip address 192.168.1.1/31
ip ospf network point-to-point
ip router ospf GOLF area 0.0.0.0
# Repeat for all interfaces
interface Loopback0
ip address 135.135.135.135 255.255.255.255
ip ospf GOLF area 0
# DCIs can learn reachability to all ACI TEP IPs via
OSPF
router ospf GOLF
router-id 135.135.135.135

5
 1b
GOLF Device Configuration (Nexus 7000)
Setup VXLAN, BGP
router bgp 3 BGP EVPN and VPNv4 Configuration
router-id 135.135.135.135
! Fabric side
neighbor 192.168.1.101
remote-as 65501
update-source loopback0
interface nve1 VXLAN ebgp-multihop 10
no shutdown address-family l2vpn evpn
source-interface loopback0 send-community extended
host-reachability protocol bgp import vpn unicast reoriginate
unknown-peer-forwarding enable neighbor 192.168.1.102
vni assignment downstream all  remote-as 65501
! update-source loopback0
fabric forwarding switch-role dci-node border ebgp-multihop 10
vxlan udp port 48879 address-family l2vpn evpn
send-community extended
import vpn unicast reoriginate
! WAN side
neighbor 210.210.210.210
remote-as 3
description VPNv4 Peering with WAN-PE-Router
update-source loopback10
address-family vpnv4 unicast
send-community both
import l2vpn evpn reoriginate

6
 1c
GOLF Device Configuration (Nexus 7000)
VRF Automation Profile
https://wiki.cisco.com/display/HMMPI/Interconnect+Policy+Provisioning+%28IPP%29

configure profile vrf-common-mpls-l3vpn-dc-edge VRF profile

vrf context $vrfName
vni $include_vrfSegmentId
rd auto
address-family ipv4 unicast
route-target import $include_client_import_ipv4_bgpRT_1 evpn
route-target export $include_client_export_ipv4_bgpRT_1 evpn
route-target import $include_client_import_ipv4_bgpRT_2 evpn
route-target export $include_client_export_ipv4_bgpRT_2 evpn
route-target import $include_client_import_ipv4_bgpRT_3 evpn
route-target export $include_client_export_ipv4_bgpRT_3 evpn
route-target import $include_client_import_ipv4_bgpRT_4 evpn
route-target export $include_client_export_ipv4_bgpRT_4 evpn Add the same entries also
route-target import $include_client_import_ipv4_bgpRT_5 evpn
route-target export $include_client_export_ipv4_bgpRT_5 evpn without ‘evpn’ at the end
route-target import $include_client_import_ipv4_bgpRT_6 evpn
route-target export $include_client_export_ipv4_bgpRT_6 evpn
route-target import $include_client_import_ipv4_bgpRT_7 evpn
route-target export $include_client_export_ipv4_bgpRT_7 evpn
route-target import $include_client_import_ipv4_bgpRT_8 evpn
route-target export $include_client_export_ipv4_bgpRT_8 evpn
router bgp $asn
vrf $vrfName
address-family ipv4 unicast
advertise l2vpn evpn
label-allocation-mode per-vrf
address-family ipv6 unicast
advertise l2vpn evpn
label-allocation-mode per-vrf
interface nve $nveId
member vni $include_vrfSegmentId associate-vrf 7
 1d
GOLF Device Configuration (Nexus 7000)
Other Automation Profiles
configure profile defaultNetworkMplsL3vpnDcProfile
ipp tenant $vrfName $client_id
include profile any

MPLS L3VPN Universal profile

configure profile vrf-tenant-profile

vni $vrfSegmentId
bridge-domain $bridgeDomainId
member vni $vrfSegmentId
interface bdi $bridgeDomainId
vrf member $vrfName
ip forward
no ip redirects
ipv6 forward
no ipv6 redirects
no shutdown VRF tenant profile

8
 1e
GOLF Device Configuration (Nexus 7000)
OpFlex Peering

ipp
profile-map profile defaultNetworkMplsL3vpnDcProfile include-profile vrf-common-mpls-l3vpn-dc-edge
local-vtep nve 1
bgp-as 3
identity 135.135.135.135
fabric 1
opflex-peer 192.168.1.101 8009
opflex-peer 192.168.1.102 8009
ssl encrypted Opflex session Establishment

Spine loopback IP addresses

9
GOLF Device Configuration
ASR1000/CSR1000v

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 1a
GOLF Device Configuration (ASR1000/CSR1000v)
Basic Config

# fabric facing interface Setup infra router bgp 3 BGP EVPN Configuration
interface g1.4 connectivity bgp router-id 135.135.135.135
bgp log-neighbor-changes
no shutdown neighbor 192.168.1.101 remote-as 65501
encapsulation dot1q 4 neighbor 192.168.1.101 ebgp-multihop 10
ip address 192.168.1.1 255.255.255.252 neighbor 192.168.1.101 update-source Loopback0
ip ospf network point-to-point neighbor 192.168.1.102 remote-as 65501
neighbor 192.168.1.102 ebgp-multihop 10
neighbor 192.168.1.102 update-source Loopback0
# Repeat for all the interfaces !
address-family l2vpn evpn
interface Loopback0 import vpnv4 unicast
neighbor 192.168.1.101 activate
ip address 135.135.135.135 255.255.255.255 neighbor 192.168.1.101 send-community both
ip ospf 100 area 0 neighbor 192.168.1.102 activate
neighbor 192.168.1.102 send-community both
# DCIs can learn reachability to all ACI exit-address-family
TEP IPs via OSPF
router ospf 100
router-id 135.135.135.135
nsr
network 192.168.1.1 0.0.0.0 area 0
11
 1b
GOLF Device Configuration (ASR1000/CSR1000v)
Basic Config

vxlan udp port 48879 VXLAN

!
interface nve1
no ip address
source-interface Loopback0
host-reachability protocol bgp
vxlan udp port 48879
no shutdown

opflex agent OpFlex Agent

service vxlan-evpn
nve-id 1
bdi-ip 100.1.1.1 255.255.255.0
domain Fabric1
identity dci-[135.135.135.135]
peer Spine1 ip-address 192.168.1.101 tcp-port 8009 src-ip-address 135.135.135.135
peer Spine2 ip-address 192.168.1.102 tcp-port 8009 src-ip-address 135.135.135.135

12
GOLF Device Configuration
ASR9000

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 1a
GOLF Device Configuration (ASR9000)
Basic Config
route-policy pass-all BGP EVPN and VPNv4 Configuration
pass
end-policy
# fabric facing interface Setup infra !
connectivity router bgp 3
interface ten1/1/0.4 bgp router-id 135.135.135.135
encapsulation dot1q 4 address-family l2vpn evpn
ipv4 address 192.168.1.1 255.255.255.252 ! Fabric side
neighbor 192.168.1.101
remote-as 65501
# Repeat for all the interfaces update-source Loopback0
ebgp-multihop 10
interface Loopback0 address-family l2vpn evpn
import stitching-rt re-originate
ip address 135.135.135.135 255.255.255.255 route-policy pass-all in
ip ospf 100 area 0 encapsulation-type vxlan
route-policy pass-all out
advertise vpnv4 unicast re-originated stitching-rt
# OSPF peering with the spines ! Configure other Spines as EVPN Neighbors
router ospf 100 ! WAN side
router-id 135.135.135.135 neighbor 210.210.210.210
remote-as 3
area 0 update-source Loopback0
interface ten1/1/0.4 address-family vpnv4 unicast
! import reoriginate stitching-rt
advertise vpnv4 unicast re-originated
<list other interfaces>

14
 1b
GOLF Device Configuration (ASR9000)
Basic Config

interface nve1 VXLAN

source-interface Loopback0
vxlan-udp-port 48879
no shutdown

dci-fabric-interconnect OpFlex Agent

auto-configuration-pool
bgp-as 3
bridge-group bg1
bd-pool 1 1001
bvi-pool 1 1001
vni-pool 1 1001
local-vtep nve 1
!
fabric 1001
opflex-peer 192.168.1.101
opflex-peer 192.168.1.102
!
identity 135.135.135.135

15
ACI and GOLF Integration
Summary of Configuration Steps

• The following steps are required to ensure a successful ACI and GOLF
integration
1. Properly configure the GOLF devices
One time Day-0
2. Create Fabric Access Policies
configuration
3. Create the L3Out in ‘infra’ Tenant
4. Create the L3Out in ‘Overlay’ Tenant
Recurring configuration
5. ‘Overlay’ Tenant VRF Configuration
6. Verification of Config Pushed on GOLF Device

16
Fabric Policies
Configure D-TEP Anycast Address

Note: this must be configured in the “Multi-Pod”

section even for a single Fabric scenario.

17
 2a
Fabric Access Policies
Create External Routed Domain

VLAN Tag between ACI Spines

and IPN devices

18
 2b
Fabric Access Policies
Spine Profile

Select the Spine Nodes

Associate the Interface

profile

19
 2c
Fabric Access Policies
Interface Profile
Select interfaces connecting to
IPN devices

Link to the previously created

AAEP (and External L3
Domain)

20
ACI and GOLF Integration
Summary of Configuration Steps

• The following steps are required to ensure a successful ACI and GOLF
integration
1. Properly configure the GOLF devices
One time Day-0
2. Create Fabric Access Policies
configuration
3. Create the L3Out in ‘infra’ Tenant
4. Create the L3Out in ‘Overlay’ Tenant
Recurring configuration
5. ‘Overlay’ Tenant VRF Configuration
6. Verification of Config Pushed on GOLF Device

21
 3a
L3Out in ‘infra’ Tenant
Create L3Out and Enable Control Planes

Enable BGP and OSPF

22
 3b
L3Out in ‘infra’ Tenant
Create Logical Node Profiles

Spine nodes

Spines northbound interfaces

(connecting to GOLF or IPN)

GOLF devices as EVPN peers

23
 3c
L3Out in ‘infra’ Tenant
Set the Provider Label

Set the Provider Label

24
 3d
L3Out in ‘infra’ Tenant
VLAN Encap for Logical Interface Profiles

Ensure VLAN 4 is configured

25
 3e
L3Out in ‘infra’ Tenant
Associate OSPF Interface Policy Name

Ensure VLAN 4 is configured

26
 3f
L3Out in ‘infra’ Tenant
Details of OSPF Interface Policy Name

27
 3g
L3Out in ‘infra’ Tenant
Verification of Day-0 Configuration
• At the end of Day-0 configuration, the GOLF device should have successfully
established OSPF peerings with the spines (or intermediate IPN devices) and MP-
BGP EVPN peerings with the spines
Pod1-GOLF1# sh ip ospf neighbors
OSPF Process ID 100 VRF default
Total number of neighbors: 3
Neighbor ID Pri State Up Time Address Interface
192.168.1.101 1 FULL/ - 16:06:50 192.168.3.1 Eth1/2.4
192.168.1.102 1 FULL/ - 16:06:49 192.168.3.5 Eth1/1.4

Pod1-GOLF1# sh bgp l2vpn evpn summary

BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 203.203.203.203, local AS number 3
BGP table version is 145, L2VPN EVPN config peers 2, capable peers 2
8 network entries and 10 paths using 832 bytes of memory
BGP attribute entries [3/432], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down

State/PfxRcd
192.168.1.101 4 65501 1021 1005 145 0 0 16:06:41 2
28
192.168.1.102 4 65501 1023 1007 145 0 0 16:06:38 2
ACI and GOLF Integration
Summary of Configuration Steps

• The following steps are required to ensure a successful ACI and GOLF
integration
1. Properly configure the GOLF devices
One time Day-0
2. Create Fabric Access Policies
configuration
3. Create the L3Out in ‘infra’ Tenant
4. Create the L3Out in ‘Overlay’ Tenant
Recurring configuration
5. ‘Overlay’ Tenant VRF Configuration
6. Verification of Config Pushed on GOLF Device

29
 4a
L3Out in ‘Overlay’ Tenant
Match ‘infra’ Tenant Label

No need to enable any routing protocol

Match the label previously configured on

the L3Out for the Infra Tenant

No need to configure Node and Interfaces

Profiles

30
 4b
L3Out in ‘Overlay’ Tenant
Define External Networks

Define the external networks that can

communicate to the fabric

31
 4c
L3Out in ‘Overlay’ Tenant
Associate L3Out to the BD

Associate the L3Out to the BD(s)

32
ACI and GOLF Integration
Summary of Configuration Steps

• The following steps are required to ensure a successful ACI and GOLF
integration
1. Properly configure the GOLF devices
One time Day-0
2. Create Fabric Access Policies
configuration
3. Create the L3Out in ‘infra’ Tenant
4. Create the L3Out in ‘Overlay’ Tenant
Recurring configuration
5. ‘Overlay’ Tenant VRF Configuration
6. Verification of Config Pushed on GOLF Device

33
 5a
‘Overlay’ Tenant VRF Configuration
Fabric BGP Route Targets, VRF on WAN Edge

Define the RTs for the Tenant VRF

34
 5b
‘Overlay’ Tenant VRF Configuration
Enable OpFlex for the Tenant VRF

Enable OpFlex and define the name of

the VRF pushed to the GOLF devices
via OpFlex

35

‘Overlay’ Tenant VRF Configuration
Host Routes Advertisement Configuration (Optional)
APIC Configuration
 Host route advertisement is enabled at the VRF level
 Applies to all the public subnets that are advertised via the
L3Out connection
5c
GOLF Router Configuration
 Spines assign the L2VNI to the Ethernet Tag ID when
originating type-2 route for a given host route/endpoint
 VTEPs normally expect the Ethernet Tag ID to be zero, so by
default the GOLF router would discard those Type-2
advertisements
 Additional knob has been added to change this default
behavior:

router bgp 3
router-id 111.111.111.111
address-family l2vpn evpn
allow-vni-in-ethertag

Configuration needed on
all the GOLF routers
36
ACI and GOLF Integration
Summary of Configuration Steps

• The following steps are required to ensure a successful ACI and GOLF
integration
1. Properly configure the GOLF devices
One time Day-0
2. Create Fabric Access Policies
configuration
3. Create the L3Out in ‘infra’ Tenant
4. Create the L3Out in ‘Overlay’ Tenant
Recurring configuration
5. ‘Overlay’ Tenant VRF Configuration
6. Verification of Config Pushed on GOLF Device

37
 6a
OpFlex Automation
Verification of Config Pushed to GOLF Device (Nexus 7000)

POd1-GOLF1# show ipp fabric

Global info:
config-profile defaultNetworkMplsL3vpnDcProfile
include-config-profile vrf-common-mpls-l3vpn-dc-edge
local-vtep nve 1
bgp-as 3
identity 203.203.203.203

Fabric 1 (Healthy)
opflex-peer 192.168.1.101:8009 (Connected and ready)
opflex-peer 192.168.1.102:8009 (Connected and ready)
ssl encrypted

Tenant Policies
1: Fabric Vrf: GOLF-Tenant1:T1-Web, Vrf: GOLF-T1
RT v4:(1:1,1:1) v6:(nil,nil)
Id 7, HostId: 7
flags 0x0

framework_p: 0xf02006bc

38
 6a
OpFlex Automation
Verification of Config Pushed to GOLF Device (Nexus 7000)
N7K-N77-GOLF# sh system internal config-profile applied-config database
config attributes:
vni 1504097
bridge-domain 2000
member vni 1504097
interface Bdi2000
vrf member GOLF-T1
ip forward
no ip redirects
ipv6 forward
no ipv6 redirects
no shutdown Note: this configuration (with the
ipp tenant GOLF-T1 7
vrf context GOLF-T1 exception of the “ipp tenant” command)
vni 1504097
rd auto could be manually applied in
address-family ipv4 unicast
route-target import 1:1 evpn deployments not using OpFlex for
route-target export 1:1 evpn
route-target import 1:1 automation
route-target export 1:1
router bgp 3
vrf GOLF-T1
address-family ipv4 unicast
advertise l2vpn evpn
label-allocation-mode per-vrf
address-family ipv6 unicast
advertise l2vpn evpn
label-allocation-mode per-vrf
interface nve1
member vni 1504097 associate-vrf 39