#CLUS

Multicast in ACI

John Weston, Technical Marketing Engineer

Data Center Networking
BRKACI-2608

#CLUS
Session Objectives

At the end of the session, the participants should be able to:

 Understand how tenant multicast traffic is forwarded in the ACI fabric

Initial assumption:
 The audience already has good knowledge of ACI main concepts
(Tenant, BD, EPG, L3Out, Multi-Pod, Multi-Site, etc.)
 The audience already has a good understanding of multicast
routing (IGMP, PIM)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• Multicast Data-Plane
• Multicast Control-Plane
• Layer-2 IP Multicast
• Layer-3 Multicast
• Multicast Configuration
• Multicast in a Multi-Pod Fabric
• Multicast in a Multi-Site Architecture
• Inter-VRF Multicast

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-2608

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
ACI Multicast Features Timeline
• Layer-3 Multicast support
(Cloudscale ASICs)
• External RP
• Auto-RP/BSR
• ASM
• SSM • L3 Multicast support for FEX
• IGMP/PIM filtering features (No L3out) • MLD snooping
2014 2016 2018 2019
1.0(1) 2.0(1) 2.0(2) 3.1(1) 4.0(1) 4.1(1)
2016 2017 • L3 Multicast with Multi-Site
• Layer-2 Multicast • Multicast with Multi-Pod • Inter-VRF multicast
• IGMP snooping • RP in the fabric (PIM anycast)*
• Fabric IGMP snooping querier • Multicast Route Scale (32k Mroutes)**
• External querier
• 8,000 multicast group scale
(IGMP snooping)
*RP in the fabric not supported with Multi-Site
**FX only
#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
ACI
Multicast Data-plane
 Review

Layer-2 Unicast VXLAN packet

= VTEP (VXLAN Tunnel Endpoint)

 ACI is a VXLAN fabric

 Unicast packets sent across the
VTEP VXLAN IP Payload fabric will be encapsulated in a
TEP 10.0.48.64 TEP 10.0.48.67 unicast VXLAN packet
 The outer source and destination IP
addresses are VXLAN tunnel
BD1 VNID: 16383905
BD1
endpoints
BD1
 The inner packet carries tenant data
EP-A EP-B

Src VTEP IP Dst VTEP IP BD VNID

10.0.48.64 10.0.48.67 16383905 EP-B MAC EP-A MAC EP-A IP EP-B IP

Outer Outer Outer Inner Inner

VXLAN Src IP Dst IP Payload
SRC IP DST IP UDP Dst MAC Src MAC

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 Review

Layer-3 Unicast VXLAN packet = VTEP (VXLAN Tunnel Endpoint)

VRF1
VNID 2129923

VTEP VXLAN IP Payload BD1

VNID 16383905
TEP 10.0.48.64 TEP 10.0.48.67

BD2
VNID 15728630

BD1 BD2

EP-A EP-B

Src VTEP IP Dst VTEP IP VRF VNID

10.0.48.64 10.0.48.67 2129923 EP-A IP EP-B IP

Outer Outer Outer Inner Inner

VXLAN Src IP Dst IP Payload
SRC IP DST IP UDP Dst MAC Src MAC

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 Multicast VXLAN encapsulation

• Multicast traffic is also

encapsulated in VXLAN across
the fabric.
• Tenant multicast traffic (inner
Outer Inner
Multicast VXLAN Multicast Payload
packet) is encapsulated in an
outer VXLAN multicast packet
BD1 BD1 BD2

EP1 EP2
10.10.10.10 10.10.10.11
Multicast Source Receiver for
group 239.1.1.1

Note: Also used for other multi-destination BUM traffic (Broadcast, unknown unicast, multicast)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Layer-2 IP Multicast VXLAN packet
= VTEP (VXLAN Tunnel Endpoint)

ACI Multicast terminology

 GIPi: Group IP Inner address. Multicast address in the
VTEP VXLAN IP Payload
inner VXLAN packet. Tenant multicast traffic running in
the tenant VRF
 GIPo: Group IP Outer address. Multicast address in the
BD1 VNID: 16383905 outer VXLAN packet. This is the multicast address
BD1
BD GIPo: 225.0.44.144 used for distributing multicast traffic across the fabric
BD1

EP-A EP-B
Multicast Source Receiver joined 239.1.1.1

Src VTEP IP Dst BD GIPo BD VNID Multicast GIPi

EP-A EP-A IP
10.0.48.64 225.0.44.144 16383905
MAC 239.1.1.1
MAC

Outer Outer Outer Inner Inner

VXLAN Src IP Dst IP Payload
SRC IP DST IP UDP Dst MAC Src MAC

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Layer-3 Multicast VXLAN packet
= VTEP (VXLAN Tunnel Endpoint)

VRF1 APIC allocates a GIPo

VNID 2129923 to all BDs and
GIPo: 225.1.192.96
multicast enabled
VTEP VXLAN IP Payload BD1 VRFs
VNID 16383905
GIPo: 225.0.44.144

BD2
VNID 15728630
GIPo: 225.1.170.32
BD1 BD2

EP-A EP-B
Multicast Source Receiver 239.1.1.1

Src VTEP IP Dst VRF GIPo VRF VNID GIPi

10.0.48.64 225.1.192.96 2129923 EP-A IP 239.1.1.1

Outer Outer Outer Inner Inner

VXLAN Src IP Dst IP Payload
SRC IP DST IP UDP Dst Eth Src Eth

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 ACI Unicast and Multicast VXLAN Summary
Single pod and Multi-Pod fabrics*
Tenant Layer-2 Unicast Packet
Outer
SRC VTEP DST VTEP BD VNID Dst MAC Src MAC Src IP Dst IP Payload
UDP

Tenant Layer-3 Unicast Packet

Outer Fabric Fabric
SRC VTEP DST VTEP VRF VNID Src IP Dst IP Payload
UDP MAC MAC

Tenant Layer-2 Multicast Packet

Outer Mcast
SRC VTEP BD GIPo BD VNID Src MAC Src IP GIPi Payload
UDP MAC

Tenant Layer-3 Multicast Packet

Outer Fabric Fabric
SRC VTEP VRF GIPo VRF VNID Src IP GIPi Payload
UDP MAC MAC

*Multisite, remote leaf, and vPod use Head end replication to send multicast

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Multicast
Control-Plane
Multicast Underlay Control-Plane
Underlay Control-Plane
• ACI uses the GIPo range 225.0.0.0/15
configured during fabric setup by
default (configurable)
• The underlay multicast groups are
IS-IS
separate from tenant multicast groups
• APIC assigns GIPo addresses to BDs
and VRFs
• PIM is not used in the underlay
• GIPo groups are advertised using IS-IS
GM-LSPs

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Multicast Underlay Control-Plane
FTAGs (Forwarding Tags)
FTAG Root for FTAG Root for FTAG Root for FTAG Root for
Underlay Control-Plane
Tree 1, 5, 9 Tree 2, 6, 10, Tree 3, 7, 11, Tree 4, 8, 12
FTAG 1 FTAG 3  IS-IS is used to build loop free
distribution trees called FTAG trees.
R R
 All FTAG trees extend to all nodes
(loop free topology)
 Multiple trees achieve load balancing
across the fabric for multicast traffic
 There are 16 FTAG trees but only 12
trees are used for user traffic

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Multicast Overlay Control-Plane
IGMP/PIM/COOP
Overlay Control-Plane
• IGMP is used by hosts to join multicast
group (version 2, version 3 supported)
• COOP is used within the fabric for
COOP COOP
advertising multicast interest to spines
and border leaves
• PIM is used on L3Out connections and
IGMP
IGMP Bridge Domain SVIs
PIM

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Multicast Overlay Control-Plane
PIM Modes

PIM • Border leaves run full

PIM protocol and peer
with external PIM
BD1 BD3
BD1 L3Out L3Out routers
RPIM RPIM
RPIM RPIM
• PIM enabled L3Outs
PIM are required for L3
PIM
PIM
multicast even if
• Non-border leaves run PIM on source and receivers
bridge domain SVIs are inside the fabric

• PIM on non-border leaves runs

in passive mode and will not
peer with any external devices

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 GIPo and FTAG selection
 The GIPo address allocated to a BD or
VRF will be a /28 address. The last four
bits of this address are used for the FTAG
selection
 The actual address sent in the VXLAN
packet will be one of the addresses in the
/28 range

225.1.22.208
BD1 BD1

EP-A EP-B
1 1 0 1 0 0 0 0

FTAG selection

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 GIPo and FTAG selection
 When the source leaf sends a multicast
Multicast flow hash
selects FTAG 1.
FTAG 1 VXLAN packet it selects an FTAG using
Sends multicast
VXLAN packet to
the last four bits in the last octet of the
group 225.1.122.209 R destination multicast address (GIPo)

225.1.22.209

1 1 0 1 0 0 0 1
EP1 EP2
10.10.10.10 10.10.10.11
Multicast Source Receiver for
group 239.1.1.1
FTAG 1

GIPo: 225.1.22.208
FTAG range: 225.1.22.209-220

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 GIPo and FTAG selection
 When the source leaf sends a multicast
Multicast flow hash
selects FTAG 2. FTAG 2
VXLAN packet it selects an FTAG using
Sends multicast
VXLAN packet to
the last four bits in the last octet of the
group 225.1.122.210 R destination multicast address (GIPo)

225.1.22.210

1 1 0 1 0 0 1 0
EP1 EP2
10.10.10.10 10.10.10.11
Multicast Source Receiver for
group 239.1.1.1
FTAG 1

GIPo: 225.1.22.208
FTAG range: 225.1.22.209-220

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Multicast GIPo Usage
 Multicast is also used in the overlay for other multi-destination traffic. Non-IP
multicast, Spanning tree, Broadcast, Unknown Unicast. The BD GIPo is used for
other multi-destination traffic

Non-Multicast Routing Enabled BD Multicast Routing Enabled BD

Broadcast BD GIPo BD GIPo

Unknown
Unicast BD GIPo BD GIPo
Flood

Multicast BD GIPo VRF GIPo

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Layer-2
IP Multicast
Layer-2 Multicast
• In this section layer-2 multicast refers to IP multicast packets
forwarded on a layer-2 network segment (BD/subnet)
• It is not Layer-2 non-IP multicast packets. (multicast packets with
a destination multicast MAC address without an IP header)
• Also excludes link local multicast (224.0.0.0/24). Link local
multicast is always forwarded to all ports in the BD

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Layer-2 IP multicast forwarding Overview

Supported from release 1.0

Layer-2 multicast is always contained within the bridge domain

Forwarded using the BD GIPo address

Layer-2 multicast is supported on all generation leaf and spine

switches

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Use case: Layer-2 multicast in bridge domain
Multicast sources and receivers in the same subnet (bridge domain)
No L3 multicast requirement

• Multicast will be forwarded to all

leaf switches where the BD is
deployed using the BD GIPo
BD GIPo
• Each Leaf switch will receive the
multicast stream and forward
out front panel ports based on
Bridge Domain
the BD Unknown Multicast
configuration
EP1 EP2 • Flood
10.10.10.10 10.10.10.11
Multicast Source Receiver for • Optimized Multicast Flood
group 239.1.1.1

Multicast Receiver
source sending sends IGMP
multicast report

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 BD Flood Modes
Multicast sources and receivers in the same bridge domain

Unknown Multicast: Flood Multicast is “unknown” when there

are no IGMP snooping entries for
the group

eth1/1 eth1/2 eth1/1 eth1/2 eth1/1 eth1/2

EP1
10.10.10.10
Multicast Source
Bridge Domain

Unknown multicast is forwarded out all ports in the BD

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 BD Flood Modes
Multicast sources and receivers in the same bridge domain

Unknown Multicast: Flood If a group is known (IGMP snooping table

entry) streams for that group will be forwarded
based on the IGMP snooping table (per leaf)

eth1/1 eth1/2 eth1/1 eth1/2 eth1/1 eth1/2

IGMP Join
10.10.10.10 Receiver for
Multicast Source group 239.1.1.1

Bridge Domain

Known multicast forwarded based on IGMP snooping table (per leaf)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 BD Flood Modes
Multicast sources and receivers in the same subnet (bridge domain)

Unknown Multicast: Optimized Flood Optimized flood forwards only based

on IGMP snooping table across all leaf
switches where BD is deployed

eth1/1 eth1/2 eth1/1 eth1/2 eth1/1 eth1/2

IGMP Join
10.10.10.10 Receiver for
Multicast Source group 239.1.1.1

Bridge Domain

Unknown multicast is forwarded based on IGMP snooping table

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Which BD mode to use?
Bridge Domains without PIM enabled

• Optimized Flood is similar to IGMP snooping where multicast is only

sent to interested receivers.
• Requires an IGMP snooping querier to trigger report messages from
hosts
• Flood will forward multicast out all ports in the BD. Receivers will
receive multicast in the absence of a querier but can consume
bandwidth and result in unnecessary flooding of multicast traffic

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
IGMP Snooping Querier
Configure an IGMP snooping Enable querier under the BD
1 2
policy for the BD. Enable the subnet
querier option

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Use case: Layer-2 multicast with external querier
Bridge domain is Layer-2 only with gateway outside of fabric
• Fabric supports external
querier
Leaf# show ip igmp snooping querier
Vlan IP Address Version Expires Port • A port that receives an IGMP
12 10.1.1.1. v2 00:04:12 Po1
query or PIM hello will
leaf# show ip igmp snooping mrouter become an IGMP snooping
Type: S - Static, D - Dynamic, V - vPC Peer Link
I - Internal, F - Fabricpath core port
mrouter port
U - User Configured
Vlan Router-port Type Uptime Expires • The external queries will be
12 Po1. D 00:15:24 00:03:36 forwarded out all BD ports
• IGMP reports and all
multicast traffic will be sent
BD1 BD1 BD1 Po1
BD1 BD1 BD1 IGMP Query
out mrouter ports
PIM Hello

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
What about IPv6?
• MLD snooping support added in release 4.1(1)
• Supports MLDv1 and MLDv2*
• Supports up to 2,000 IPv6 multicast groups across the fabric
• Not supported on 1st generation leaf switches (EX/FX/FX2 only)
• Supports the same flooding modes as IPv4 (Flood/Optimized Flood)
• IPv6 Multicast routing (PIMv6) not currently supported (roadmap)

* MLDv2 forwarded in hardware based on (*,G) lookup

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Layer-3 Multicast
Layer-3 Multicast
Scale, SW/HW Support

HW Scale
Supported spines
• All supported spines • Supports up to 8,000
Mroutes (fabric wide
Supported Leaf
• N93180YC-EX and per leaf) Default
• N93108TC-EX profile
• N93180LC-EX
• N93180YC-FX • Supports up to
• N93108TC-FX 32,000 Mroutes (FX
• N9348GC-FXP only) High Dual Stack
• N9336C-FX2
• N93240YC-FX2
profile
• N9358GY-FXP
• N93360YC-FX2
• N93216TC-FX2

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Layer-3 Multicast Overview
• Supported Modes • Inter-VRF multicast 4.0
PIM-SM (ASM)
Multi-Site support
•
• 4.0
• PIM-SSM
• PIM and IGMP filters
• External Rendezvous Point (RP)
• Static • Multicast traffic not filtered by
• Auto-RP contracts
• BSR • Not supported with remote leaf
• Fabric RP (Anycast RP) 4.0
• No Auto-RP or BSR
• Not supported with Multi-Site

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Layer-3 Multicast BD Flood/OMF and IGMP snooping

• IGMP snooping must be enabled for PIM enabled BDs

• Multicast packets will only be forwarded out BD ports if there is an IGMP
snooping entry
• It is not required to change the L3 Unknown Multicast Flooding setting
(this differs from non-PIM enabled BDs)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 Layer-3 Multicast: Role of L3Outs

• L3Outs are a requirement for multicast

routing
• The L3Out must be configured with a
loopback address and have PIM enabled
L3Out must
have loopback • PIM is supported on L3Outs with routed or
routed sub-interfaces and includes Layer

L3Out L3Out
port-channels
RPIM RPIM
BD3 BD3
RPIM RPIM • Not supported on L3Outs with SVI
interfaces
Multicast Multicast
Source Receiver

L3Out is required even when • For ASM the L3Out provides a path to
source and receiver are inside the external RP or can function as the RP
the fabric (4.0 feature)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 Layer-3 Multicast: External RP

External RP Options
Source leaf BL is always • Static RP (supports multiple RPs)
is FHR LHR for fabric
PIM Register • Auto-RP (MA Filters)

• BSR (BSR Filters)

L3Out L3Out
RPIM RPIM
IGMP BD3 BD3 PIM Join
Multicast
Join Multicast
Source Receiver

RP

Source IP for
Register packets

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Layer-3 Multicast: Fabric RP

Fabric RP
• Anycast RP (equivalent to PIM
anycast, does not run MSDP)

RP RP RP RP • Does not support peering anycast

RP with external RP
L3Out L3Out L3Out L3Out
RPIM RPIM RPIM RPIM
BD3 BD3 BD3 BD3
• All PIM enabled border leaves will
become anycast RP members (per
VRF)

• Required for inter-VRF multicast

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Layer-3 Multicast: Stripe Winners
The COOP database maintains
multicast group interest in the fabric
and is distributed to all active border
leaves in the fabric

One BL will be selected as the stripe

winner for a multicast group and will
be the designated forwarder for the
group
COOP
COOP
G1 G2
L3Out L3Out L3Out L3Out

IGMP PIM Join RPIM

BD3
RPIM
BD3 PIM Join
IGMP
G1:239.10.10.10 G2: 239.11.11.11
Join Multicast Join
Multicast
Receiver Receiver
G1:239.10.10.10 G2: 239.11.11.11

RP

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Layer-3 Multicast: Fast Convergence

Fast Convergence is a feature where

all border leaves (stripe winners and
non stripe winners) will send joins for
(*,G) or (S,G). Only the stripe winner
will forward the multicast traffic into
the fabric

COOP
COOP
G1 G2
L3Out L3Out L3Out L3Out

IGMP PIM Join RPIM

BD3
RPIM
BD3 PIM Join
IGMP
G1:239.10.10.10 G2: 239.11.11.11
Join Join
Multicast Multicast G2: 239.11.11.11 G1: 239.10.10.10
Receiver Receiver
G1:239.10.10.10 G2: 239.11.11.11

RP/Source

External network has multicast state.

Improves convergence time after a failure

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Use Case: Layer-3 Multicast
Source and Receiver in the Fabric (ASM)
ACI switches do not continuously
BD1 GIPo send register packets to the RP. Only
225.1.22.208 a single packet is sent to the RP. The
BD2 GIPo leaf will send periodic null registers to
225.0.163.208
the RP.
VRF GIPo The border leaf seeing that the
225.1.192.64
source is local to the fabric (RPF is
COOP
VRF GIPo fabric interface) will send a PIM prune
225.1.192.64
towards the RP
Register

L3Out L3Out
BD1 BD1 BD2 BD2 RPIM RPIM
RPIM RPIM RPIM RPIM BD3 BD3
IGMP
Join

10.10.10.10 Receiver for

PIM
group 239.1.1.1
Multicast Join
Source
VRF1 RP

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
BDs without PIM enabled in multicast enabled VRF
Multicast in BD without PIM enabled
 IGMP Querier (fabric or external)
 OMF mode

BD3 GIPo
225.0.163.208
BD3 GIPo
225.0.163.208

L3Out L3Out
BD1 BD1 BD3 BD3 BD2 BD2 RPIM RPIM
RPIM RPIM RPIM RPIM BD3 BD3

Multicast Multicast
Source Receiver

VRF1 RP

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Use Case: Layer-2 Multicast with PIM enabled BDs
Source and Receiver in the Fabric Same BD

BD1 GIPo
225.1.22.208
VRF GIPo
225.1.192.64

VRF GIPo
225.1.192.64

L3Out L3Out
BD1 BD1 BD1 BD1 RPIM RPIM
RPIM RPIM RPIM RPIM BD3 BD3
IGMP
Join

10.10.10.10 Multicast
Multicast Receiver PIM
Source Join
VRF1 RP

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Layer-2 IP Multicast in PIM enabled Bridge Domains
Layer-2 multicast forwarding
behavior for PIM enabled BDs
• All Layer-2 and Layer-3
multicast will be routed
• TTL will be decremented
VXLAN
SRC MAC
BD1 MAC
TTL
2
SRC IP
10.10.10.10
DST IP
239.1.1.1
Payload
Payload
VXLAN
SRC MAC
BD1 MAC
TTL
1
SRC IP
10.10.10.10
DST IP
239.1.1.1
Payload
Payload
twice. Once on the ingress
node and once on the egress
Decrement TTL
BD1 VRF1 BD1 Decrement TTL node (regardless of number
of transit nodes)

SRC MAC TTL SRC IP DST IP Payload SRC MAC TTL SRC IP DST IP Payload
• RP must be defined for ASM
EP1 MAC 3 10.10.10.10 239.1.1.1 Payload BD1 MAC 1 10.10.10.10 239.1.1.1 Payload

EP1
EP2 • Source MAC will be rewritten
10.10.10.11
10.10.10.10
Multicast Source
Receiver for to BD MAC
group 239.1.1.1

*Excludes link local multicast (224.0.0/24)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Use Case: Layer-3 Multicast
Source and Receiver in the Fabric (ASM)

BD1 GIPo
225.1.22.208
VRF GIPo
225.1.192.64

Decrement TTL
VRF GIPo
Decrement TTL
225.1.192.64

L3Out L3Out
BD1 BD1 BD1 BD1 RPIM RPIM
RPIM RPIM RPIM RPIM BD3 BD3

10.10.10.10 Multicast
Multicast Receiver
Source
VRF1 RP

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Use Case: Layer-3 Multicast
PIM-SSM

BD1 GIPo PIM-SSM Support

225.1.22.208
BD2 GIPo
• Default SSM multicast range
225.0.163.208
232.0.0.0/8
VRF GIPo
225.1.192.64
• Configuration of different
VRF GIPo
COOP
225.1.192.64 SSM range supported
• PIM enabled BDs run
BD1 BD1 BD2
L3Out L3Out IGMPv2. Must enable
BD2 RPIM RPIM
RPIM RPIM RPIM RPIM BD3 BD3 IGMPv3 in BD IGMP policy
IGMPv3
Join • Supports IGMP SSM
10.10.10.10
Multicast
Receiver for
group 239.1.1.1 Translate (allows IGMP
Source 10.10.10.10
Source
VRF1 version 2 hosts to join SSM
groups)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Multicast in a
Multi-Pod Fabric
 ACI Multi-Pod – Underlay Control-Plane
IPN Configured PIM Bidir for
fabric GIPo range
vrf context ipn Spines send IGMP joins for
ip pim rp-address 10.101.200.1 group-list 225.0.0.0/15 bidir
BD and VRF GIPo groups
towards IPN
IPN
IGMP
IGMP
PIM BiDir

APIC Cluster

BD1 BD2 L3Out L3Out L3Out L3Out BD2 BD1

RPIM RPIM RPIM RPIM RPIM RPIM RPIM RPIM

Multicast Multicast
Receiver Receiver
Group1: Group2:
239.10.10.10 239.20.20.20 Core

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
ACI Multi-Pod – Overlay Control-Plane
BGP EVPN Type 6 route

IPN

PIM BiDir

APIC Cluster
COOP COOP COOP

BD1 BD2 L3Out L3Out L3Out L3Out BD2 BD1

RPIM RPIM RPIM RPIM RPIM RPIM RPIM RPIM
PIM PIM
(*,G) join (*,G) join
Multicast
Receiver
Multicast
Receiver
239.10.10.10 239.20.20.20
Group1: Group2:
239.10.10.10 239.20.20.20 Core

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
ACI Multi-Pod – Data-plane
Multicast Forwarding between pods – Data Plane

IPN
Spine 2 selected Spine 2 selected
authoritative for
PIM BiDir authoritative for
VRF GIPo VRF GIPo

APIC Cluster

BD1 BD2 L3Out L3Out L3Out L3Out BD2 BD1

RPIM RPIM RPIM RPIM RPIM RPIM RPIM RPIM

Multicast Multicast Multicast

Receiver Receiver Source
Group1: Group2:
239.10.10.10 239.20.20.20 Core

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Multicast
Configuration
How do we configure multicast routing?
Three (or four) steps to
enable L3 multicast
1. Enable Multicast at the VRF
2. Enable multicast for BDs
where Multicast is required
3. Enable multicast for the L3Out
4. Configure and RP (for ASM)

1 Enable Multicast at the VRF level

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Enable multicast for BDs and L3outs

2 Enable PIM for 3 Enable PIM for

Bridge Domains L3out

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Configure PIM Rendezvous Point (RP)
4 Configure an RP address (ASM)

Fabric RP Configuration

Auto-RP and BSR options are

also supported

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Multicast Features
Multicast routing is enabled by enabling the VRF, BD, L3out and RP
configuration for ASM but additional multicast features are supported
IGMP Features PIM Features
PIM Authentication
IGMP Report Policy PIM timers
IGMP static join PIM Join/Prune filters
IGMP fast leaves PIM neighbor filters
IGMP state-limit PIM multicast domain boundary
IGMP SSM translate Auto-RP
BSR

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Multicast with
Multi-Site Fabrics
 L3 Multicast with Multi-Site Fabrics Control-Plane
No Multicast Control-Plane between sites

Inter-Site Network

COOP COOP
L3Out L3Out Site 1 Site 2 L3Out L3Out
RPIM RPIM RPIM RPIM

PIM PIM
Must use external RP. RP Site 1 and 2 connect to
in the fabric not supported RP
the common external RP
with Multi-site via local L3Outs

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 L3 Multicast with Multi-Site Fabrics Data-Plane
Multicast traffic is sent across sites using
HER (Head-End Replication). Sent to the
multicast data-plane TEP
Inter-Site Network

L3Out L3Out Site 1 Site 2 L3Out L3Out

RPIM RPIM RPIM RPIM

RP

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 Layer-2 Multicast over Multi-Site (Supported since 3.0)
• Stretched BDs with Intersite BUM Traffic Enabled (no PIM configuration required)
• Within a site the multicast will be sent to the BD GIPo multicast address (unique per site)
• Multicast across sites will be sent in the HREP tunnel to the remote site multicast TEP address
• Spine elected as Designated Forwarder (DF) replicate the stream to each remote sites where the BD is stretched
• At the receiving spine the multicast will be sent down the FTAG tree to the receiving site BD GIPo multicast address

Inter-Site
Site 2 Multicast TEP: 10.100.102.200
Network

BD1 VNID  16711545

HREP tunnel dest: 10.100.102.200

Site 1
BD1 VNID  16514962
BD1 GIPo  225.0.195.240

Site 2
BD1 VNID  16711545
BD1 GIPo  225.1.128.160
BD1 BD1 BD1

Site 1 Source Receiver Receiver Site 2

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 Layer-3 Multicast Routing with Multi-Site
L3 Multicast over Multi-Site (Source Inside the Fabric)
• Built as Routing-First Approach (decrement TTL at source and destination ACI leaf node)

• L3 Multicast is always sent to the VRF GIPo within a site (existing behavior)

• Between sites it is sent over the HREP tunnel to the Multicast TEP of the remote sites where the VRF is stretched (the VXLAN
header will include the source site VRF VNID)
• L3 Multicast at the receiving site will be sent in the VRF GIPo of the receiving site

Inter-Site
Site 2 Multicast TEP: 10.100.102.200
Network

Site1 VRF VNID  2293762

HREP tunnel dest: 10.100.102.200

Site 2 Decrement TTL

VRF VNID
VRF Decrement TTL VRF GIPo  225.1.248.16
Site 1
L3Out VRF VNID 2293762 L3Out
VRF VRF GIPo  225.1.248.32 VRF

BD1 BD2 Decrement TTL BD1 BD2

VRF VRF
Site 1 Site 2
RP Source VRF VRF
Source Receiver

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 Layer-3 Multicast Routing with Multi-Site
L3 Multicast over Multi-Site (Source Inside the Fabric)
• Built as Routing-First Approach (decrement TTL at source and destination ACI leaf node)

• L3 Multicast is always sent to the VRF GIPo within a site (existing behavior)

• Between sites it is sent over the HREP tunnel to the Multicast TEP of the remote sites where the VRF is stretched (the VXLAN
header will include the source site VRF VNID)
• L3 Multicast at the receiving site will be sent in the VRF GIPo of the receiving site

Inter-Site
Site 2 Multicast TEP: 10.100.102.200
Network

Site1 VRF VNID  2293762

HREP tunnel dest: 10.100.102.200

Site 2 Decrement TTL

VRF VNID
VRF Decrement TTL VRF GIPo  225.1.248.16
Site 1
L3Out VRF VNID 2293762 L3Out
VRF VRF GIPo  225.1.248.32 VRF

BD1 BD2 Decrement TTL BD1 BD2

VRF VRF
Site 1 Site 2
RP Source VRF VRF
Source Receiver

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Layer-3 Multicast Routing with Multi-Site
L3 Multicast over Multi-Site (Source Outside the Fabric)
• Local L3Out must be used to receive traffic from an external source
• Multicast traffic from external sources dropped on the spines (to avoid traffic duplication)

Multicast traffic from external

Multicast traffic from external Inter-Site sources is dropped on spine. Not
sources is dropped on spine. Not
sent over HREP tunnels Network sent over HREP tunnels

Site 2
VRF VNID
VRF GIPo  225.1.248.16
Site 1
VRF
L3Out VRF VNID 2293762 L3Out
VRF VRF GIPo  225.1.248.32 VRF

BD1 BD2 BD1 BD2

VRF VRF
Site 1 Site 2
RP Source VRF VRF
R1 R2 R3

Source
#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Inter-VRF
multicast
Inter-VRF Multicast

Allows multicast receivers in one VRF to receive multicast traffic from sources in another VRF

Always forwarded in the source VRF across the fabric

Receiving switch responsible for crossing VRFs

The RPF lookup for the source/RP is done in the source VRF

Requires source VRF to be present on all switches where receiver VRF is located

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Inter-VRF Multicast Flow
VRF1 VRF2 Note: The source VRF is not
BL11
automatically programmed on the
switch where the receiver VRF is
20.1.1.10 30.1.1.30 located. Requires user to configure
source an EPG on the switch in the source
multicast receiver
VRF
239.23.1.1

Multicast sent across fabric

in source VRF. (VRF1
Source VRF must be
GIPo/VRF1 VNID)
deployed on all leaf
switches where there
are receivers

L3out L3out L3out L3out

VRF1 VRF2 VRF1 VRF2

VRF1 VRF2
multicast packets arrive
VRF1
at the egress leaf in the
Source Receiver source VRF and are Receiver
forwarded to receivers
in the receiver VRF

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Inter-VRF Multicast Flow RP Requirements
VRF1 VRF2
Requires RP in the
fabric configured in Static RP configured
the source VRF 20.1.1.10 30.1.1.30
in receiver VRF
source
multicast receiver
Fabric RP: 239.23.1.1
123.1.1.1 RP: 123.1.1.1

Anycast RP in source VRF

RP 123.1.1.1

L3out L3out L3out L3out

VRF1 VRF2 VRF1 VRF2

VRF1 VRF1 VRF2

Source Receiver Receiver

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Inter-VRF Multicast source VRF not deployed on
leaf with receivers: Not Supported
VRF1 VRF2 BL11

30.1.1.30
multicast receiver

If the source VRF is not

deployed on the switch
multicast traffic will not be
received

L3out L3out L3out L3out

VRF1 VRF2 VRF1 VRF2

VRF1 VRF2

Source Receiver Receiver

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Inter-VRF Source VRF only deployed where
required: Supported
VRF1 VRF2 BL11

20.1.1.10 30.1.1.30

RP:123.1.1.1 multicast receiver

Source VRF does not

need to be deployed
Receiver VRF does not on all leaf switches.
need to be deployed on Only where there are
source leaf receivers in other VRFs

L3out L3out L3out

VRF2 VRF1 VRF2

VRF1 VRF1 VRF2 VRF2 VRF2

no
Source Receiver no multicast receivers multicast
Receiver
receivers

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Inter-VRF Configuration

Fabric RP in source VRF

Configure route-map with

multicast group range and
the VRF where the multicast
sources are located

Note: sources for a multicast group cannot be in different VRFs

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Layer-3 Multicast
Flows
Source Inside, Receiver Inside
Receiver sends IGMP

3
2
Spine sends COOP
Leaf sends COOP message to BL with
message to spine with multicast interest.
multicast interest. COOP COOP (*,G) state created on
(*,G) state created on BL
leaf

BD1 BD3
BD2 L3Out L3Out
RPIM RPIM LHR
IGMP
PIM

4
Source Receiver
EP1 EP2 BL sends PIM (*,G)
join towards RP.
RP
10.10.1.10 10.11.1.20
Multicast Source Multicast Receiver (*,G) state created on
1 RP (*,G)
Receiver sends IGMP
Join to leaf

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Source Inside, Receiver Inside
Source sends multicast

4
Leaf sends multicast
traffic across fabric on
Leaf connected to VRF GIPo PIM register
source sends register
to RP. (S,G) created 2
on leaf

BD1 BD3
BD2 L3Out L3Out
FHR RPIM 5 RPIM
Multicast data traffic
sent to all leaf
Leaf forwards
switches where VRF
Multicast traffic multicast
is deployed
based on PIM
IGMP
1 Source Receiver (*,G) prune
snooping table
EP1 EP2
Multicast sender 10.10.1.10 10.11.1.20 PIM RP
starts sending Multicast Source Multicast Receiver
multicast traffic register stop (*,G)
3

Note: Step 2 and Step 4 occur almost simultaneously. The If there are no
external receivers the
FHR Leaf forwards multicast data traffic on the VRF GIPo as RP will send register
soon as it sends the PIM register stop and (*,G) prune
towards source

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Source Inside, Receiver Outside
Source sends multicast

4 6
Leaf sends multicast BL adds OIF for
traffic across fabric on interface towards
Leaf connected to VRF GIPo PIM register 5
receiver
source sends register Router connected to
to RP. (S,G) created receiver joins the SPT
on leaf
2
tree and send (S,G)
join toward source
BD1 BD3
BD2 L3Out L3Out PIM
FHR RPIM RPIM
Multicast data traffic
sent to all leaf (S,G) join
switches where VRF
Multicast traffic is deployed
PIM
1 Source (*,G) prune Multicast
Receiver
EP1
Multicast sender PIM
starts sending
10.10.1.10
Multicast Source
register stop
RP LHR
multicast traffic
3
RP sends register
stop and (*,G) prune
towards source

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Source Outside, Receiver Inside
Receiver sends IGMP

3
2
Spine sends COOP
Leaf sends COOP message to BL with
message to spine with multicast interest.
multicast interest. COOP COOP (*,G) state created on
(*,G) state created on BL
leaf

BD1 BD3
BD2 L3Out L3Out
RPIM RPIM
IGMP LHR
PIM Multicast
4 (*,G) Join Source
Receiver
EP2 BL sends PIM (*,G)
join towards RP.
RP
10.11.1.20
Multicast Receiver (*,G) state created on
1 RP (*,G)
Receiver sends IGMP
Join to leaf

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Source Outside, Receiver Inside
Source sends multicast data

5
LHR BL switches over to
SPT and sends (S,G)
join towards source

BD1 BD3
BD2 L3Out L3Out
RPIM
4
RPIM
3 PIM
(S,G) Join
FHR
Leaf forwards BL forwards multicast
multicast traffic on VRF GIPo to
based on all leaf switches in
VRF Multicast Multicast
IGMP Source
snooping table
Receiver traffic
EP2 PIM
RP
10.11.1.20
Multicast Receiver
register
2 (*,G)
1

RP sends multicast Source starts sending

traffic towards B L multicast data. RP
where (*,G) originated learns of source (via
register from FHR)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
 Source Outside, Receiver Outside
Receiver sends IGMP

2 3
Spine sends COOP
BL sends COOP
message to BL with
message to spine with
multicast interest. COOP COOP multicast interest.
(*,G) state created on
(*,G) state created on
egress BL
ingress BL 4
BL sends (*,G) join
L3Out L3Out
BD1
L3Out L3Out
PIM towards RP
PIM RPIM RPIM RPIM RPIM
(*,G) join (*,G) join

1
Multicast
LHR connected to
LHR
Source
receiver sends (*,G)
Join towards RP
Multicast
Receiver
RP
(*,G)

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Source Outside, Receiver Outside
Source sends multicast data

7
5 4
BL sends (S,G) join
Egress BL receives BL receives multicast towards source
multicast and on L3out and
forwards out OIF forwards on VRF GIPo
towards receiver tree
2
FHR sends register
L3Out
towards RP
L3Out
BD1
L3Out L3Out
PIM
PIM RPIM RPIM RPIM RPIM
(S,G) join
(S,G) join
1
6 3 Multicast Multicast Source sends
traffic Source multicast data
LHR receives RP forwards multicast PIM
RP register
FHR
Multicast
multicast traffic and Receiver traffic towards BL
switches over to (shared tree) (*,G)
shortest path tree.
Sends (S,G) join

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS BRKACI-2608 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Thank you

#CLUS
#CLUS