BRKDCN 2099

#CiscoLive
Multi-Tier Fabrics
Network Designs for the Modern Data Center
Lukas Krattiger, Distinguished Engineer

@CCIE21921
BRKDCN-2099
#CiscoLive
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here
4 Enter messages/questions in the Webex space
Webex spaces will be moderated https://ciscolive.ciscoevents.com/ciscolivebot/#BRKDCN-2099
by the speaker until June 17, 2022.
#CiscoLive BRKDCN-2099 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Abstract
Have you ever asked yourself what "Clos" is or where that
Leaf/Spine thing comes from? If yes, this is the right session for you.
We are going to cover Fat-Tree, Clos and Leaf/Spine designs and
expand beyond just the Spine layer. We will spend some time on the
Super-Spine and even Super-Spine fabrics. How you can cost
effectively use 100G/400G and where fixed vs. modular Switches
make sense.
Introduction
• A brief Overview on Leaf and Spine Topologies
• Some terms and Nomenclature
• Design and Sizing considerations
• The 3-Stage and 5-Stage Clos
• Or how we build beyond the Leaf and Spine Topology
• Super-Spine and Spine Planes
• How does this fit into your DataCenter
Companion Sessions – Week at a Glance
BRKDCN-2267 BRKDCN-2913 BRKDCN-2563
Bits and Bytes
External Connectivity VXLAN BGP EVPN A Day in the Life of a

for VXLAN EVPN Multi-Site VXLAN EVPN Packet
(10:30am-11:15am) (10:30am-11:15am) (8am-8:45am)
BRKDCN-1999 BRKDCN-2012 BRKDCN-3238 BRKDCN-2345

Introduction to VXLAN vPC with VXLAN Overlay Multicast in A Day in the Life of a
(8am-8:45am) (1pm-1:45pm) VXLAN EVPN Multi-Site Packet
(1pm-1:45pm) (9:30am-10:15am)
Monday Tuesday Wednesday Thursday

Control, Design

Introduction to NDFC Automating DCs Operate vPC with Streaming Telemetry
Operation,
(9:30am-10:15am) with NDO and NDFC VXLAN EVPN (1pm-1:45pm)

(1pm-1:45pm) (1pm-1:45pm)

Multi-Tier Fabric RFC5549 in the DC Deploying Nexus Elastic Service
(1pm-1:45pm) (4pm-4:45pm) Dashboard Redirection
(2:30pm-3:15pm) (1pm-1:45pm)
• Introduction
• Paradigm and Fundamentals
Agenda • Design Evolution
• Conclusion
Paradigm and
Fundamentals
The Paradigm
• A Leaf and Spine Topology S S S S
L L L L L L
BRKDCN-2099 © 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
The Paradigm
Spine
• A Leaf and Spine Topology S S S S
• Variations or Names of the

same:
• Fat Tree
• Folded Clos
L L L L L L
• 3 Stage Clos
• 2 Tier Network Leaf
The Paradigm
Spine
• A Leaf and Spine Topology Stage #2 S S S S
• 3 Stages
• 2 Tiers
Stage #1 Stage #3
L L L L L L
Leaf
The Paradigm
Spine
• A Leaf and Spine Topology 2nd Tier S S S S
• 3 Stages
• 2 Tiers
1st Tier L L L L L L
Leaf
The Paradigm
More Bandwidth
• A Scale Out Architecture S S S S
• More Leaf = More Ports

• More Spine = More Bandwidth
L L L L L L
More Ports
The Paradigm
• N+1 Redundancy S S S S
• Redundancy increases by
Building out the Topology
• On Spine failure
• 4 Spine = 25% impact
L L L L L L
• 8 Spine = 12.5% impact
The Paradigm
N
W E
S
• Modern Application Needs S S S S
• Every (1) North to South

Connection, requires eight (8)
East to West
• User Access the Frontend
(Web)
L L L L L L
• Frontend connects to App, DB,
Storage etc.
The Paradigm
N
W E
S
• Optimized for East to West S S S S
• Consistent Latency from Leaf to

Leaf
• Wide ECMP
• Flexibility for North to South

• External Connectivity at Leaf or L L L L L L
Spine layer
The Paradigm
• Bandwidth Requirements S S S S
• Oversubscription
User to Server
Bandwidth
L L L L L L
Server to Server Bandwidth
How Many Spines do I need?
It Depends
Oversubscription and Maximum Redundancy as the Criteria
S S S S
Host Attachment
Requirements
• 48 Server per Rack
• 2x 25Gbps NIC per Server
• 1x NIC per Switch
L L L L L L
Rack1 Rack2 Rack3
S S S S
Host Attachment Resulting Uplink Requirements
Requirements • 48x 25Gbps per Leaf
• 48 Server per Rack No Oversubscription (1:1) • 1.2Tbps Uplink from Leaf to
• 2x 25Gbps NIC per Server Spine
• 1x NIC per Switch • 12x 100Gbps towards Spine
L L L L L L
Rack1 Rack2 Rack3
12 Spines
S S S S
Host Attachment Resulting Uplink Requirements
Requirements • 48x 25Gbps per Leaf
• 48 Server per Rack No Oversubscription (1:1) • 1.2Tbps Uplink from Leaf to
• 2x 25Gbps NIC per Server Spine
• 1x NIC per Switch • 12x 100Gbps towards Spine
L L L L L L
Rack1 Rack2 Rack3
Fabric Size – 12 Spine, 1:1 Oversubscription
12 Spines
Let’s Do some Math
S S S S Spine
8 Slot Modular Chassis
36x 100Gbps Port per Linecard

No Oversubscription (1:1)
Total: 288 Spine Ports
Leaf
288 Spine Ports = 288 Leaf

Switch
48x 25Gbps Host Ports Per Leaf

L L L L L L Total: 13’828 Host Ports
Fabric Bandwidth
1:1 Oversubscription
1.2Tbps Uplink * 288 Leaf
Total: 345.6Tbps
Rack1 Rack2 Rack3
Modular vs. Fixed for Single Fabric (2 Tier)
12 Spines
S S S S Spine
8 Slot Modular Chassis Fixed Spine
36x 100Gbps Port per Linecard 64x 400Gbs

Total: 288 Spine Ports Total: 256 Spine Ports
Leaf
288 Spine Ports = 288 Leaf 256 Spine Ports = 256 Leaf Switch
Switch
48x 25Gbps Host Ports Per Leaf 48x 25Gbps Host Ports Per Leaf
L L L L L L Total: 13’828 Host Ports Total: 12’288 Host Ports
Fabric Bandwidth
1:1 Oversubscription 1:1 Oversubscription
1.2Tbps Uplink * 288 Leaf 1.2Tbps Uplink * 256 Leaf
Total: 345.6Tbps Total: 307.2Tbps

Rack1 Rack2 Rack3
PRO PRO
More Leaf Less Latency
More Ports Less Power
More Bandwidth All Single ASIC
Replacing the Chassis Size
(8 Slot to 16 Slot Chassis)
Is Scale Finite !?!?!?
12 Spines
S S S S Spine
8 Slot Modular Chassis 16 Slot Modular
36x 100Gbps Port per Linecard 36x 100Gbps Port per Linecard
Leaf
288 Spine Ports = 288 Leaf 576 Spine Ports = 576 Leaf
Switch Switch
Fabric Bandwidth
Total: 345.6Tbps Total: 691.2Tbps

Rack1 Rack2 Rack3
Doubling the Host Port Scale

Replacing the Spine Port Speed
(100Gbps to 400Gbps)
Is Scale Finite !?!?!?
12 Spines
S S S S Spine
8 Slot Modular Chassis 8 Slot Modular Chassis
36x 100Gbps Port per Linecard 36x 400Gbps Port per Linecard
Leaf
288 Spine Ports = 288 Leaf 1152 Spine Ports = 1152 Leaf
Switch Switch
Fabric Bandwidth
Total: 345.6Tbps Total: 1’382.4Tbps

Rack1 Rack2 Rack3
Quadrupling the Host Port Scale (Breakout 4x 100Gbps at Spine)

Scale is very Linear in 2 Tier
Networks
More Spine Ports Results in More … Fabric Bandwidth, Leaf Count, Host
Ports
Attributes to Scale
S S S S S S S S S S S S
L L L L L L L L L L L L L L L L L L
288 Leaf 576 Leaf 1152 Leaf
8 Slot Modular 16 Slot Modular 8 Slot Modular

36x 100Gbps 36x 100Gbps 36x 400Gbps
1:1 Oversubscription 1:1 Oversubscription 1:1 Oversubscription
13’828 Host Ports 27’648 Host Ports 55’296 Host Ports
Scale-Up to Faster
Scale-Up to Fill Chassis Scale-Up to Bigger Chassis
Linecards
Oversubscription Ratio doesn’t influence Host Port scale

• How to Scale 2 Tier Networks
• Every Spine scaling involves
Scale-Up
• Initial Chassis Build Out (4)
• Add Linecards (up to 4)
• Increase Chassis Size (4 > 8)
Points To Think • Add more Linecards (up to 8)
About
• Add more Linecards (up to 16)
• Increase Chassis Speed (4)
• Add Faster Linecards (up to 4)
BRKDCN-2099 Not Bad, Just Scale-Up

© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
What Else to Think About?
12 Spines
S S S S
• What is my Failure Domain?
• What is my Change Domain?
• What is my Overall Scale?
• What is my Fabric Solution Scale?
• What is my Fabric SLA?
• What is my Maximum Downtime?
L L L L L L
1152
Leaf
What did I really Build – Untangling the Details
2 Tier / 5 Stage Network with Modular Spine
Modular Modular Modular Modular
Chassis Chassis Chassis Chassis
F F F F F F F F F F F F
S S S S
L L L L L L L L L L L L
What you think you Built What you really Built

2 Tier Leaf and Spine Network (3 Stage) 2 Tier Leaf and Spine Network (5 Stage)
Spine: Modular Chassis (4 Slot, 8, Slot, 16 Slot) Spine: Modular Chassis (4 Slot, 8, Slot, 16 Slot)
Leaf: Fixed Switch (single ASIC) Leaf: Fixed Switch (single ASIC)
What did I really Build – Untangling the Details
Stage #3 F F F F F F F F F F F F
Tier #2 S S S S
Stage #2 L L L L L #4 L
Stage L L L L L L
Tier #1 Stage #1 Stage #5

What you think you Built What you really Built

2 Tier Leaf and Spine Network (3 Stage) 2 Tier Leaf and Spine Network (5 Stage)
Spine: Modular Chassis (4 Slot, 8, Slot, 16 Slot) Spine: Modular Chassis (4 Slot, 8, Slot, 16 Slot)
Leaf: Fixed Switch (single ASIC) Leaf: Fixed Switch (single ASIC)
Forwarding Behavior
• Fixed Leaf represents 1 Stage
• Modular Spine represents 3 Stages
L L L L L L
What you really Built

2 Tier Leaf and Spine Network (5 Stage)
Spine: Modular Chassis (4 Slot, 8, Slot, 16 Slot)
Leaf: Fixed Switch (single ASIC)
Latency Behavior
1μs F F F F F F F F F F F F
1μs
1μs L L L L L L L L L L L L
• Generally, all Modular Switches
operate in Store-and-Forward (SnF)
• Packet Size dependent Latency
• Without Speed Change, Leaf operates
in Cut-Through
1μs 1μs
• Packet Size independent Latency
L L L L L L • Normalized, difference in Latency from
Spine (Modular) to Leaf (Fixed) is 3:1
What you really Built
2 Tier Leaf and Spine Network (5 Stage)
Leaf: Fixed Switch (single ASIC)
Intra-Chassis Behavior
• Within Leaf Tier and Between Leaf and
Spine Tier
• Full Behavior / Protocol Control
• Layer-3 ECMP Load Balancing
• Standards-based Routing Protocols
• BFD for Fast Failure Detection
• Minimal Exposure for Brownout
L L L L L L
• Within Spine Tier
What you really Built • Intra-Chassis Load Balancing
2 Tier Leaf and Spine Network (5 Stage) • Intra-Chassis Protocol
Leaf: Fixed Switch (single ASIC) • Intra-Chassis Failure Detection
• Fully Redundant Components
With all this Knowledge, why do we
still build 2 Tier Scale Up Networks
with Big-Fat Spines?
The Elephant in the Room
Design Evolution
The Journey to Build Better and Further
S S S S S S S S
S S S S S S S S
F F F F F F F F
L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L
2 Tier Leaf Spine 3 Tier Leaf-Spine-SuperSpine 3 Tier Leaf-Fabric-Spine

(5 Stages) (11 Stages) (5 Stages)
The Status Quo
Discussed at Length
• A perfectly valid way S S S S
• Tends to have “Finite Scale”

• Maximum Chassis capacity
• Maximum Speed per Port
L L L L L L
• Many Locations of Redundancy

• Redundant Chassis Components 2 Tier Leaf Spine
(5 Stages)
• Condensed Link and Bandwidth
Presence •
•
Use Modular Chassis at Spine
Use More Density on Linecards
• Aggregated within a Chassis • Use Higher Bandwidth per Port
S S S S S S S S
S S S S S S S S
F F F F F F F F

• Scale-Out; Introduce a 3rd Tier • To Infinity and the Beyond

• Interconnect multiple 2 Tier “PODs”
Let’s Move Forward • Use Modular or Fixed Spine &
SuperSpine
• Use High Port Density
• Use High Bandwidth per Port
Nothing New
Let’s Not Stop Here
• Avoiding Scale-Up with another Tier S

S
S
S
S
S
S
S
• Distributed Link and Bandwidth S S S S S S S S
Presence
• Disaggregated across Tiers
• Increases the “Finite Scale”
• No Dependency on Chassis capacity or
3 Tier Leaf-Spine-SuperSpine
Speed per Port (11 Stages)
• Many Locations of Redundancy • Scale-Out; Introduce a 3rd Tier

• Interconnect multiple 2 Tier “PODs”
• Redundant Chassis Components • Use Modular or Fixed Spine &
SuperSpine
• Allows for Cost Optimization •
•
Use High Port Density
Use High Bandwidth per Port
S S S S S S S S
S S S S S S S S
F F F F F F F F

• To Infinity and the Beyond

Let’s Move Forward
What we learned from the Cloud Titans
Building Scalable DataCenter Networks
#1 #2 #3 #4
Simplicity is Key Scale as you Go Fail but Fail Fast Redundant and Repeatable
Simple Design Principals Scale is Never Finite Reduce Brown-Out Exposure Risk is Never an Option
How the Cloud
Titans Build
• Increasing Scale-Out in all Tiers
S S S S
• Reduce to the Max S S S S
• Simple Design Principles

F F F F F F F F
• Increases the “Finite Scale”

• Scale as You Go
• Disaggregated Redundancy
• Flexible Link and Bandwidth Distribution 3 Tier Leaf-Fabric-Spine
(5 Stages)
• Further Possibility for Cost Optimization
• To Infinity and the Beyond
Step #1 – Don’t Build Fabric for Maximum Leaf
• Fixed Switch at the Fabric (Tier #2)
• Depending on Oversubscription Ratio, reserve Ports
• 1:1 Oversubscription
• Reserve 50% from Tier #2 to Tier #3
• Common Fixed Spine Options
• 64x 100Gbps (6.4Tbps Single ASIC)
F F F F
L L L L L L
128 Leaf
Fabric #1
Step #1 – Don’t Build Fabric for Maximum Leaf
• Fixed Switch at the Fabric (Tier #2)
• Depending on Oversubscription Ratio, reserve Ports
• 1:1 Oversubscription
• Reserve 50% from Tier #2 to Tier #3
• Common Fixed Spine Options
Tier #2: Nexus 9364C-GX2B - 64x Ports 400Gbps
50% Uplink to 3rd Tier (32x 400Gbps)
50% Downlink for Leaf (128x 100Gbps)
Breakout: 32x 400Gbps = 128x 100Gbps
Tier #2 F F F F
L L L L L L
128 Leaf
6’144 Ports
Fabric #1
Step #2 – Repeat for Host Port Scale (Scale Out)
• Increasing Fabrics at need
• of Host Port
• of Oversubscription between Tier #2 and #3
• Result Defines Tier #2 to Tier #3 Uplinks
• and respectively Tier #3 Requirements
F F F F F F F F F F F F F F F F
L L L L L L L L L L L L L L L L L L L L L L L L
128 Leaf 128 Leaf 128 Leaf 128 Leaf

6’144 Ports 6’144 Ports 6’144 Ports 6’144 Ports
Fabric #1 Fabric #2 Fabric #3 Fabric #4
Step #3 – Designing Tier #3
• Introducing Tier #3 Planes
• Blue, Green, Yellow, Red
• Rule: Tier #2 Blue only connects to Tier#3 Blue
Tier #3 S S S S

Step #3 – Designing Tier #3
• Inter-Fabric Decision is made at Leaf Layer
• Deterministic Path from Tier #2 to Tier #3
• Rule: Once entered a Plane, you stay in the Plane
Tier #3 S S S S

Step #3 – Designing Tier #3 (Single Link)
Tier #3: Nexus 9316-GX - 16x Ports 400Gbps
4x 1x 400Gbps = 1.6Tbps inter-Fabric Bandwidth
Tier #3 S S S S
4x 1x 400Gbps

Step #3 – Designing Tier #3 (Single Link)
• This Setup gets you all the way to 16 Fabrics
• 16x 6’144 Host Ports = 98’304 Host Ports
Tier #3: Nexus 9316-GX - 16x Ports 400Gbps • Valid with VXLAN EVPN
4x 1x 400Gbps = 1.6Tbps inter-Fabric Bandwidth • 16 Sites (max Site scale is 128)
Tier #3 S S • S BGW at Tier #2 S
• Tier #3 is Layer-3 transport Network
4x 1x 400Gbps

Step #3 – Designing Tier #3 (Multi-Link)
• Or increase Bandwidth to 6.4Tbps
• Multiple Links (4) between Tier #2 and Tier #3
Tier #3 S S S S
4x 4x 400Gbps

Step #3 – Designing Tier #3 (Multi-Link)
• Or increase Bandwidth to 12.8Tbps
• Multiple Links (8) between Tier #2 and Tier #3
Tier #3 S S S S
4x 8x 400Gbps

Step #4 – Increasing the Tier #3 Planes
Tier #3: Nexus 9316-GX (4 per Plane) - 16x Ports 400Gbps
S S S S
S S S S
S S S S
Tier #3 S S S S
4x 4x 400Gbps

Step #4 – Increasing the Tier #3 Planes
• This Setup gets you all the way to 16 Fabrics
Tier #3: Nexus 9316-GX (4 per Plane) - 16x Ports 400Gbps • 16x 6’144 Host Ports = 98’304 Host Ports
4x 4x 400Gbps = 6.4Tbps inter-Fabric Bandwidth • Valid with VXLAN EVPN
S S • 16 Sites (max
S Site scale is 128)S
S S S S
Tier #3 S
S
S
S • S BGW
S at Tier #2 S
S
• Tier #3 is Layer-3 transport Network
4x 4x 400Gbps

S S S S S S S S
S S S S S S S S
F F F F F F F F

• Use of Modular Spine • Use of Modular Spine and SuperSpine • Use of Fixed Fabric and Spine
• 4-, 8- or 16-Slot Chassis • 16-Slot Chassis • 16x to 64x 400Gbps per Fabric (Tier #2)
• 36x 100Gbps (or 36x 400Gbps) Linecard • 36x 100Gbps Linecard • 16x to 64x 400Gbps per Spine (Tier #3)
• 100Gbps Uplink between Leaf and Spine • 100G Uplink Between Leaf, Spine and • 100Gbps Uplink between Leaf and Spine
• 144 Leaf (6’912 Host Ports) SuperSpine • 400Gbps Uplink between Tier #2 and Tier #3
• 288 Leaf (13’824 Host Ports) • 512 Leaf (24’576 Host Ports) • 1 Fabric, 128 Leaf (6’144 Host Ports)
• 576 Leaf (27’648 Host Ports) • 4 Fabric, 128 Leaf (24’ 576 Host Ports)
• 1152 Leaf (55’296 Host Ports • 16 Fabric, 128 Leaf (98’304 Host Ports)
• 128 Fabric, 128 Leaf (786’432 Host
Ports)
Conclusion
We can scale from Small to Very
Large – Don’t be shy starting with a
Small Setup; we can Evolve!
Key Takeaway #1
Bigger is not Always Better; Using
Fixed-Form factor Switches is a
Modern Practice
Key Takeaway #2
More Switches != Higher Cost
Key Takeaway #3
Technical Session Surveys
• Attendees who fill out a minimum of four
session surveys and the overall event
survey will get Cisco Live branded socks!
• Attendees will also earn 100 points

in the Cisco Live Game for every
survey completed.
• These points help you get on the

leaderboard and increase your chances
of winning daily and grand prizes.
Pay for Learning with
Cisco Learning Credits
Cisco Learning and Certifications (CLCs) are prepaid training
vouchers redeemed directly
From technology training and team development to Cisco certifications and learning with Cisco.
plans, let us help you empower your business and career. www.cisco.com/go/certs
Learn Train Certify

Cisco U. Cisco Training Bootcamps Cisco Certifications and
IT learning hub that guides teams Intensive team & individual automation Specialist Certifications
and learners toward their goals and technology training programs Award-winning certification
program empowers students
Cisco Digital Learning Cisco Learning Partner Program and IT Professionals to advance
Subscription-based product, technology, Authorized training partners supporting their technical careers
and certification training Cisco technology and career certifications
Cisco Guided Study Groups
Cisco Modeling Labs Cisco Instructor-led and 180-day certification prep program
Network simulation platform for design, Virtual Instructor-led training with learning and support
testing, and troubleshooting Accelerated curriculum of product,
technology, and certification courses Cisco Continuing
Cisco Learning Network Education Program
Resource community portal for Recertification training options
certifications and learning for Cisco certified individuals
Here at the event? Visit us at The Learning and Certifications lounge at the World of Solutions
• Visit the Cisco Showcase
for related demos
• Book your one-on-one

Meet the Engineer meeting
• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs
your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand
Thank you
#CiscoLive
#CiscoLive

BRKDCN 2099

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 2099

Uploaded by

Copyright:

Available Formats

#CiscoLive

Lukas Krattiger, Distinguished Engineer

4 Enter messages/questions in the Webex space

Webex spaces will be moderated https://ciscolive.ciscoevents.com/ciscolivebot/#BRKDCN-2099

by the speaker until June 17, 2022.

• How does this fit into your DataCenter

External Connectivity VXLAN BGP EVPN A Day in the Life of a

BRKDCN-1999 BRKDCN-2012 BRKDCN-3238 BRKDCN-2345

Monday Tuesday Wednesday Thursday

BRKDCN-1119 BRKDCN-2731 BRKDCN-2106 BRKDCN-2991

(9:30am-10:15am) with NDO and NDFC VXLAN EVPN (1pm-1:45pm)

BRKDCN-2099 BRKDCN-2828 BRKDCN-3714 BRKDCN-2879

• A Leaf and Spine Topology S S S S

• A Leaf and Spine Topology S S S S

• Variations or Names of the

• A Leaf and Spine Topology Stage #2 S S S S

• A Leaf and Spine Topology 2nd Tier S S S S

• A Scale Out Architecture S S S S

• More Leaf = More Ports

• Every (1) North to South

• Consistent Latency from Leaf to

• Flexibility for North to South

Server to Server Bandwidth

Rack1 Rack2 Rack3

Rack1 Rack2 Rack3

Rack1 Rack2 Rack3

8 Slot Modular Chassis

36x 100Gbps Port per Linecard

288 Spine Ports = 288 Leaf

48x 25Gbps Host Ports Per Leaf

1.2Tbps Uplink * 288 Leaf

8 Slot Modular Chassis Fixed Spine

36x 100Gbps Port per Linecard 64x 400Gbs

1:1 Oversubscription 1:1 Oversubscription

1.2Tbps Uplink * 288 Leaf 1.2Tbps Uplink * 256 Leaf

Total: 345.6Tbps Total: 307.2Tbps

8 Slot Modular Chassis 16 Slot Modular

1:1 Oversubscription 1:1 Oversubscription

1.2Tbps Uplink * 288 Leaf 1.2Tbps Uplink * 576 Leaf

Total: 345.6Tbps Total: 691.2Tbps

Doubling the Host Port Scale

8 Slot Modular Chassis 8 Slot Modular Chassis

1:1 Oversubscription 1:1 Oversubscription

1.2Tbps Uplink * 288 Leaf 1.2Tbps Uplink * 1152 Leaf

Total: 345.6Tbps Total: 1’382.4Tbps

Quadrupling the Host Port Scale (Breakout 4x 100Gbps at Spine)

288 Leaf 576 Leaf 1152 Leaf

8 Slot Modular 16 Slot Modular 8 Slot Modular

Oversubscription Ratio doesn’t influence Host Port scale

BRKDCN-2099 Not Bad, Just Scale-Up

What you think you Built What you really Built

Tier #1 Stage #1 Stage #5

What you think you Built What you really Built

What you really Built

2 Tier Leaf Spine 3 Tier Leaf-Spine-SuperSpine 3 Tier Leaf-Fabric-Spine

• A perfectly valid way S S S S

• Tends to have “Finite Scale”

• Many Locations of Redundancy

2 Tier Leaf Spine 3 Tier Leaf-Spine-SuperSpine 3 Tier Leaf-Fabric-Spine

• Scale-Out; Introduce a 3rd Tier • To Infinity and the Beyond

• Avoiding Scale-Up with another Tier S

• Distributed Link and Bandwidth S S S S S S S S

• Many Locations of Redundancy • Scale-Out; Introduce a 3rd Tier