Skywarp in Oracle Data

Centre
MPLS Profile Discussion

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Full built out

272 ToRs NCS500X
4 Spines NCS5508
1152 Leaf-Spine
links
2304 100G optics*

4 Spines

Spine Layer

NCS5508

64 NB ports
1088 SB ports

8 Slot 36x100G

* 100G optic: 100G over

1 fiber pair, 150m on
OM4 MM
4x
16
x1

BGP
MPLS ToR
E-W: 1:1 oversub

N-S: 17:1 oversub

00

Border Leaf
272x4x1
00G

Leaf Layer

NCS500X

4 NB ports on each leaf

272 Leafs

32x100G

NCS5001

40x10G:4x100G

40 SB ports on each leaf

40x10G
4x16x100G

Host Layer

272 Racks
40 Servers Single
attach

40

40

40

40

2x ASR9912 -10 Slot Chassis

8x 8X100G LCs 6 LC Slots open
Total 64 100G ports

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Interface Hu0/0/0/1

router ospf 1

ipv4 add 192.2.1.1/24

area 0

Interface Hu0/0/0/2

interface Hu0/0/0/1

ipv4 add 192.2.2.1/24

interface Hu0/0/0/2

Interface Hu0/0/0/3

interface Hu0/0/0/3

ipv4 add 192.2.3.1/24

interface Hu0/0/0/4

Interface Hu0/0/0/4
ipv4 add 192.2.4.1/24
Interface loopback 0

!
!
mpls ldp

ipv4 add 1.1.1.1/32

interface Hu0/0/0/1
interface Hu0/0/0/2
interface Hu0/0/0/3
interface Hu0/0/0/4
!

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Interface Hu0/0/0/1

router ospf 1

ipv4 add 192.2.1.2/24

area 0

Interface Hu0/0/0/2

interface Hu0/0/0/1

ipv4 add 192.2.2.2/24

interface Hu0/0/0/2

Interface Hu0/0/0/3

interface Hu0/0/0/3

ipv4 add 192.2.3.2/24

interface Hu0/0/0/4

Interface Hu0/0/0/4
ipv4 add 192.2.4.2/24

!
!

Interface loopback 0
ipv4 add 2.2.2.2/32

mpls ldp
interface Hu0/0/0/1
interface Hu0/0/0/2
interface Hu0/0/0/3
interface Hu0/0/0/4
!

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

BRCM Table
name

Scal
e

Users

Purpose

scale for MPLS

Port

NA

All
features

To enable MPLS

NA

MY_STATION_TC NA
AM

All
features

MPLS_TERMINAT NA
ION_ALLOWED

VLAN_MPLS

4K

MPLS

To enable MPLS
per VLAN

4K

MPLS_ENTRY

32K

MPLS

Stores local
label

16K (due to the

limited scale for
OL)

ECMP Group

4K

IP and
MPLS

ECMP

4K

EGR_L3_NEXT_
HOP

48K

L2, L3,
MPLS

Points to out
label and dMAC

48K

MPLS/EoM
PLS

Out label for

simple LSR, VC
label for
EoMPLS

16K

EGR_MPLS_VC_ 16K
AND_SWAP_LAB
EL_TABLE
2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

BRCM
Table
name

Scale

Users

Purpose

scale for MPLS

EGR_L3_INTF

12K

L3
features

One entry per

each MPLS
tunnel

12K (2 * configured
number of L3
interfaces)

EGR_IP_TUNN 16K
EL_MPLS

2010 Cisco and/or its affiliates. All rights reserved.

MPLS/EoM One entry per

PLS/L3VP label in the
N
label stack
(max 2 per
stack)

16K

Cisco Confidential

X Number of links from Leaf to Spine

Y Number of leafs in this Data Centre which are part of slice shown in the
topology
Expected number of IGP prefixes on leaf can be computed as per the
formula
( (X+1) * Y) + ( 1 * X) + (Y * X)
Number of paths in ECMP group X
Number of ECMP groups per prefix

For directly connected prefixes (PHP) 2

For prefixes not directly connected 1

Expected number of ECMP groups on leaf can be computed as per the

formula
( (X+1) * Y) + ( 1 * X) * 2 + (Y * X) * 2
For the given topology:
X = 4, Y = 272
Total IGP prefixes ~ 2452
Total ECMP groups ~ 3544

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

 Using ACL filter with LDP to allow only the loopback IP

addresses of other Monterreals.
This feature has been confirmed to by working by DE and
DT.
Using Ipv4 unnumbered loopback on all duplicate links.
Yet to be verified.
Filter out the directly connected prefixes in LDP to prevent
label allocation for these.
Use load sharing when we run out of ECMP groups Code
change needed

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

VRF + IPv4

Prefix

L3 DEF
IP

ECMP
Group

ECMP
ECMP
ECMP
Path
ECMP
Path
ECMP
Path
Path
Path

ECMP
ECMP
ECMP
Path
ECMP
Path
Egress
Path
Path
NH

ECMP
ECMP
ECMP
Path
ECMP
Path
VC And
Path
Path
Swap
ECMP
ECMP
ECMP
Path
ECMP
Path
Egr L3
Path
Path
Intf

Label
Table

ECMP
ECMP
ECMP
Path
ECMP
Path
Egr IP
Path
Path
MPLS

IGP Label
+ VPN
Label

Tunnel

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

 L3VPN proposed between

Leaf and Internet Gateway (ASR9K)
Leaf and Border Leaf (Havasu)
Leaf and Leaf for E-W communication
Skywarp supports per vrf VPN label only.
Each vrf would need as many VPN labels as the number of
PEs it is peering with PE:PE ratio to be understood.
Each (VPN label + IGP label combination) requires a new
ECMP group.
Current code base (leveraged from Viking doesnt share
ECMP group across prefixes with same label stack).
Need changes to share.

As Broadcom supports only one level of load balancing,

BGP ECMP cant be programmed. We will use Load sharing
here. IOW, if prefixes p1, p2, p3 all have 3 BGP paths x, y
and z p1 will use x, p2 will use y and p3 will use z and so
on.

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

10

A number of BGP paths per prefix

B number of IGP paths to reach this prefix.
C PE:PE ratio
D Total number of BGP prefixes
E total number of vrfs

Total number of ECMP groups for BGP: A * C * E

For the current topology:
A = 1, C = ?, E 8
Total usage = 8C

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

11

 2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

12