Brkini 2025

UCS Networking –
Deep Dive
David Nguyen
UCS Technical Marketing Engineer
BRKINI-2025
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
Cisco Spark spaces will be cs.co/ciscolivebot#BRKINI-2025

available until July 3, 2017.
BRKINI-2025 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• High-level system overview
• Unified Ports
• I/O module
• Fabric Interconnect Forwarding modes
• End-host mode (EHM) vs Switch mode
• Dynamic and static pinning concepts
• Server Connectivity Options
• Adapter
• NIC teaming
• Recommended designs
• Disjoint L2
• ACI
UCS LAN Details
System Overview
System Components: High-level Overview
Unified Management
Fabric Interconnect (FI)

• 10 GbE unified fabric switch
• Connects up to 160 servers
UCS Fabric Extender (FEX)

• 10 GbE unified remote linecard
• Flexible bandwidth connectivity
Compute (Blade and Rack)

• x86 industry standard
• Patented extended memory
UCS Virtual Interface Card (VIC)

• CNA adapters
• Flexible server I/O
UCS Physical Connectivity
• Server Form Factor Independent
• Blades and Racks are treated the
same
• Consolidated I/O
• Minimal cabling
• Spanning-tree Free
UCS Fabric Interconnect Portfolio
UCS 6324 (Mini) UCS 6300
UCS 6296 & 6248
UCS 6120 & 6140

Mar 2013
10GbE 40GbE
SHIPPING EOL / EOS
Fabric Interconnect Key Differences
FI 6200 series & FI 6300 series
FI 6200 Series FI 6300 Series

Features FI 6248 FI 6296 FI 6332 FI 6332-16UP
Height 1RU 2RU 1RU 1RU
Form factor Modular Modular Fixed Fixed
Physical ports 48 (using GEM) 96 (using GEM) 32 40
Max 10G ports 48 96 96* + 2** = 98 72* + 16UP = 88
Max 40G ports - - 32 24
Max FC ports 48 x 2/4/8G FC 96 x 2/4/8G FC - 16 x 4/8/16G FC
Max unified ports 48 96 - 16
Expansion module 1 x Exp module 3 x Exp modules - -
* Using 40G to 4x10G breakout cables ** Requires QSA module
IOM Key Differences
IOM 2200 series & IOM 2304
IOM 2200 Series IOM 2300

Features 2204XP 2208XP 2304
Form Factor Modular Modular Modular
Total ports 20 40 12
Max 10G ports 20 40 -
Max 40G ports - - 12
Uplink ports 4 8 4
Server ports 16 32 8
Which Fabric Interconnect?
FI 6300 Series FI 6200 Series
• 40GbE • 10GbE
• 40G FCoE • 10G FCoE
• 16G FC • 8G FC
• B-Series or C-Series • B-Series, C-Series or M-Series
• Requires UCSM 3.1(1) • UCSM 2.x(x) or 3.1(1)
FI 6324 Series
• 10GbE
• 10G FCoE
• B-Series or C-Series
• Requires UCSM 3.0
Fabric Interconnect
Port Types
• By default all ports are disabled
• LAN Uplink Port SAN
LAN
• Network ports, path to and from the network
• SAN Uplink port SAN Uplink
Appliance LAN Uplink
• SAN ports, path to SAN network Port
CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
ID
STAT
• Server Port
• Connecting to UCS servers
Server Port
• Appliance Port
• Connect to direct attach storage device i.e, NFS
Unified Ports
Dynamic Port Allocation: Lossless Ethernet or Fibre Channel
FC Eth
Native Fibre Channel Lossless Ethernet:

1/10GbE, FCoE, iSCSI, NAS
Benefits Use-cases
 Simplify switch purchase - remove  Flexible LAN & storage convergence based on
ports ratio guess work business needs
 Increase design flexibility  Service can be adjusted based on the demand
for specific traffic
 Remove specific protocol bandwidth
bottlenecks
UCS 62xx: Unified Ports
Dynamic Port Allocation: Lossless Ethernet or Fibre Channel
• Ports on the base card or the Unified Port GEM Module can either be Ethernet
or FC
• Only a continuous set of ports can be configured as Ethernet or FC
• Ethernet Ports have to be the 1st set of ports
• Port type changes take effect after next reboot of switch for Base board ports or
power-off/on of the GEM for GEM unified ports.
Base card – 32 Unified Ports GEM – 16 Unified Ports

Eth FC Eth FC
6332-16UP
Port Layout and Roles
 16 SFP+ Unified Ports  24 QSFP+ Ports

 Ports 1-16  Ports 17-40
 Ethernet or FC (4/8/16G FC / FCOE  Ethernet or FCOE
(10G)  Uplinks (Eth/FCOE/Converged)
 Uplinks (Eth/FC/FCOE/Converged)  Appliance Ports
 Appliance Ports  Server Ports
 Server Ports  Storage Ports (FCOE only)
 Storage Ports (FC/FCOE)
6332-16UP
Unified Port Groups Configuration
1 2 3
• 16 Unified Ports • 16 Unified Ports

• 16 x 4/8/16G FC ports • 16 x 4/8/16G FC ports
• Converted in blocks • Converted in blocks
• Block 1 – Ports 1-6 • Block 1 – Ports 1-6
6332 Model
40G Breakout to 4x10G
 Breakout Port Creates (4) 10Gb Ports

 Port Modes Same as on Non-Breakout Ports
 Each Port of Breakout Port Configured Individually
 Change to/from Breakout Port Requires FI Reboot
 Note: If 2nd No-Drop Class Configured with Jumbo MTU, QSFP
Breakout Ports limited to 4 (ie, 16 individual 10Gb ports) -
Remainder of QSFP Ports will be Non-Breakout 40Gb ports.
6324 Ports
10G/40G Port
Port Roles
- Uplink
- Server port
- FC/FCOE Storage port
Port Roles
- Server port
- Appliance port
- FCOE Storage port
Putting the Components Together
SAN A ETH 1 ETH 2 SAN B
MGMT MGMT
Uplink Ports
OOB Mgmt
Fabric Switch
Fabric A Fabric B
Server Ports Cluster
Fabric Extenders Chassis 1 Chassis 20

IO IO IO IO
M M M M
Virtualized Adapters A B A B
CNA CNA CNA
Compute Blades B200 B250
Half / Full width
Chassis Connectivity Options
UCS Fabric Topologies
Chassis Bandwidth Options
2208XP
only
2x 1 Link 2x 2 Link 2x 4 Link 2x 8 Links

20 Gbps per Chassis 40 Gbps per Chassis 80 Gbps per Chassis 160 Gbps per Chassis
IOM Connections
• A IOM (sometimes called ‘Fabric Extender’) provides
• 1 for internal managment
• 10G-KR sever facing links (HIF)
• Fabric links (NIF)
• The servers’ mezz cards use those IO channels for external connectivity
• Each IOM provides a separate dedicated IO channel for internal management
connectivity
UCS 2204 IO Module
Enable Dual 20 Gbps to Each Blade Server
UCS-IOM-2204XP
• Bandwidth increase for improved response esp
for bursty Applications
o 40G to the Network
o 160G to the Host Redundant
o (2x10G/ Half width slot; 4x10G/ Full
width slot)
• Latency Lowered to 0.5us within IOM
• Investment Protection with Backward and
Forward Compatibility
UCS 2208 IO Module
Enable Dual 40 Gbps to Each Blade Server
UCS-IOM-2208XP
o 80G to the Network
o (4x10G/ Half width slot; 8x10G/ Full
width slot)
• Latency Lowered to 0.5us within IOM
220x-XP Architecture
Fabric Ports to FI
2208
FLASH
2204
DRAM
Feature 2204-XP 2208-XP
EEPROM
Chassis Control ASIC Woodside Woodside

Management IO Woodside ASIC
Controller
Fabric Ports
4 8
(NIF)
Switch
Host Ports
16 32
(HIF)
Chassis 2204 CoS 8 8

Signals 2208
Internal backplane ports to blades
Latency ~ 500ns ~ 500ns
No Local Switching – ever!
Traffic goes up to FI
NIF/HIF Interfaces
• Output from “show fex detail” of the NXOS shell
<output truncated>
Fabric interface state:

Po1153 - Interface Up. State: Active
Eth1/17 - Interface Up. State: Active FI ports connecting
Eth1/18 - Interface Up. State: Active
to the FEX
Fex Port State Fabric Port
Eth1/1/1 Down Po1153
FEX/IOM Ports Eth1/1/4 Down Po1153
Link between the FI
facing towards Eth1/1/5 Up Po1153 and IOM/FEX
the server – Eth1/1/6 Down None which the server
Eth1/1/7 Up Po1153
chassis/1/port will be using
<output truncated>
UCS Internal Block Diagram
UCS 6248 UCS 6248
Fabric 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module
Interconnects
Double the Fabric Uplinks
IO Modules
2208XP 2208XP
Midplane Quadruple the Downlinks
Adapter mLOM Mezz

x16 Gen 2 x16 Gen 2
CPU 0 CPU 1
Server Blade QPI Link
UCS Blade Chassis
CLI Block diagram
• Output from “show platform software woodside sts” of the IOM
(FINAL POSITION TBD) Uplink #: 1 2 3 4 5 6 7 8
Link status: | | | |
+-+--+--+--+--+--+--+--+-+
SFP: [$][$][$][$][ ][ ][ ][ ]
+-+--+--+--+--+--+--+--+-+
| N N N N N N N N |
| I I I I I I I I |
| 0 1 2 3 4 5 6 7 |
| |
| NI (0-7) |
+------------+-----------+
|
+-------------------------+-------------+-------------+---------------------------+
| | | |
+------------+-----------+ +-----------+------------+ +------------+-----------+ +-------------+----------+
| HI (0-7) | | HI (8-15) | | HI (16-23) | | HI (24-31) |
| | | | | | | |
| H H H H H H H H | | H H H H H H H H | | H H H H H H H H | | H H H H H H H H |
| I I I I I I I I | | I I I I I I I I | | I I I I I I I I | | I I I I I I I I |
Eth1/1/1
| 0 1 2 3 4 5 6 7 | | 8 9 1 1 1 1 1 1 | | 1 1 1 1 2 2 2 2 | | 2 2 2 2 2 2 3 3 |
| | | 0 1 2 3 4 5 | | 6 7 8 9 0 1 2 3 | | 4 5 6 7 8 9 0 1 |
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+
[ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ]
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+
- - - | - | - | - | - | - - | |
1 1 1 1 1 1 1 9 8 7 6 5 4 3 2 1 Port # from the
6 5 4 3
\__\__/__/ \__\__/__/
2 1 0
\__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/ \__\__/__/
FEX Port
blade8 blade7 blade6 blade5 blade4 blade3 blade2 blade1
CLI command
• Output from “show platform software woodside rate” of the IOM
fex-1# sh platform software woodside rate

+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| Port || Tx Packets | Tx Rate | Tx Bit || Rx Packets | Rx Rate | Rx Bit |Avg Pkt|Avg Pkt| |
| || | (pkts/s) | Rate || | (pkts/s) | Rate | (Tx) | (Rx) |Err|
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
| 0-BI || 25 | 5 | 5.04Kbps || 14 | 2 | 3.09Kbps | 106 | 118 | |
| 0-CI || 65 | 13 | 20.20Kbps || 49 | 9 | 25.44Kbps | 174 | 304 | |
| 0-NI3 || 2 | 0 | 3.48Kbps || 63 | 12 | 12.98Kbps | 1072 | 108 | |
| 0-NI2 || 35 | 7 | 12.05Kbps || 6 | 1 | 2.75Kbps | 195 | 267 | |
| 0-NI1 || 3 | 0 | 3.69Kbps || 14 | 2 | 4.22Kbps | 750 | 168 | |
| 0-NI0 || 17 | 3 | 6.86Kbps || 20 | 4 | 12.38Kbps | 232 | 367 | |
| 0-HI31 || 7 | 1 | 8.00Kbps || 1 | 0 | 768.00 bps | 695 | 464 | |
Blade 1
| 0-HI30 || 5 | 1 | 1.28Kbps || 1 | 0 | 200.00 bps | 141 | 106 | |
| 0-HI23 || 5 | 1 | 1.22Kbps || 1 | 0 | 352.00 bps | 133 | 200 | |
| 0-HI19 || 13 | 2 | 2.48Kbps || 1 | 0 | 1.19Kbps | 99 | 728 | |
| 0-HI11 || 3 | 0 | 560.00 bps || 1 | 0 | 128.00 bps | 98 | 64 | |
| 0-HI7 || 4 | 0 | 616.00 bps || 1 | 0 | 560.00 bps | 77 | 332 | |
+--------++------------+-----------+------------++------------+-----------+------------+-------+-------+---+
UCS 2304 IO Module
Enable True 40 Gbps or 4x10 to Each Blade Server
UCS-IOM-2304XP
o Native 40G to the Network
• Low Latency and Over Subscription
2304 IOM Block Diagram
• Dual-Core Freescale CPU Memory
• HDMI Connector (Dongle) Mgmt

4x QSFP+
Console
• 10/100 RJ45 Mgmt CPU To FI
Flash
• Console
• 1GB DRAM FEX
• Flash Memory ASIC
Switch
32x 10G-KR
8x 40G-KR
1G IOM Cluster Link To Server
8x 100Mb
To Server (CIMC)
Fabric Ports to FI-63xx
2304 IO Module
• Interface
• NIF
• 4 x 40G QSFP
Templeton ASIC
• Connects only to FI63xx
• HIF
• 32 Interfaces
• Support 10G or 4 ports can combine to a
single 40G
1 Internal backplane ports to blades 32
No Local Switching
Fabric Link Connectivity
IO Module HIF to NIF Pinning
2208XP – 1 Link
Slot 1
HIF1-4 HIF1-4
Slot 2
NIF1 HIF5-8 HIF5-8 NIF1
Slot 3
HIF9-12 HIF9-12
Slot 4
HIF13-16 HIF13-16
Slot 5
HIF17-20 HIF17-20
Slot 6
HIF21-24 HIF21-24
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
2208XP – 2 Links
Slot 1
HIF1-4 HIF1-4
Slot 2
NIF2 Slot 3 NIF2

HIF9-12 HIF9-12
Slot 4
HIF13-16 HIF13-16
Slot 5
HIF17-20 HIF17-20
Slot 6
HIF21-24 HIF21-24
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
2208XP – 4 Links
Slot 1
HIF1-4 HIF1-4
Slot 2
NIF2 Slot 3 NIF2

NIF3 HIF9-12 HIF9-12
NIF3
NIF4 Slot 4 NIF4
HIF13-16 HIF13-16
Slot 5
HIF17-20 HIF17-20
Slot 6
HIF21-24 HIF21-24
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
2208XP – 4 Links
Slot 1
HIF1-4 HIF1-4
Slot 2
NIF2 Slot 3 NIF2

HIF9-12 HIF9-12
NIF3 NIF3
Slot 4 NIF4
NIF4
HIF13-16 HIF13-16
NIF5 Slot 5 NIF5
NIF7
Slot 6
NIF7
HIF21-24 HIF21-24
NIF8 NIF8
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
CLI Mapping
FEX Level command Interface Level command
• Output from “show fex detail” of the • Output from “show interface
NXOS shell ethernet <mod/port> fex-intf
<output truncated>
Fabric interface state:

Eth1/12 - Interface Up. State: Active Fabric FEX
Fex Port State Fabric Port Interface Interfaces
Eth3/1/1 Up Eth1/9 -----------------------------------------------
Eth3/1/2 Up Eth1/10 Eth1/9 Eth3/1/1 Eth3/1/5
Eth3/1/3 Down Eth1/11
Eth3/1/4 Up Eth1/12
Eth3/1/5 Up Eth1/9
Eth3/1/6 Down None
Eth3/1/9 Up Eth1/12
<output truncated>
What happens in a 4-link topology when you loose 1 link?
IO Module and link failure
• Fex fabric link (NIF4) fails
Slot 1
HIF1-4 HIF1-4
Slot 2
NIF1 HIF5-8 HIF5-8
NIF2 Slot 3
NIF3 HIF9-12 HIF9-12
NIF4 Slot 4
Link Failure HIF13-16 HIF13-16
Slot 5
HIF17-20 HIF17-20
Slot 6
HIF21-24 HIF21-24
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
• Fex fabric link (NIF4) fails
• Loose connectivity on adapter port Slot 1
mapped to NIF4 for blades 1 and HIF1-4 HIF1-4
5 Slot 2
NIF1 HIF5-8 HIF5-8
• If blades 1 and 5 have redundant NIF2 Slot 3
link and NIC teaming is configured NIF3 HIF9-12 HIF9-12
on the OS, then traffic will failover NIF4 Slot 4

over to the other fabric HIF13-16 HIF13-16
Slot 5
• Traffic for the other blades (1-3 HIF17-20 HIF17-20
Slot 6
and 5-7) are unaffected HIF21-24 HIF21-24
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
• Re-ack of chassis will fall back to
a 2 link topology
Slot 1
• Blades 4 and 5 will recover the port HIF1-4 HIF1-4
on the adapter that was previously Slot 2
down NIF1 HIF5-8 HIF5-8
• Blades will redistribute the NIF2 Slot 3

HIF9-12 HIF9-12
connection NIF3
Slot 4
• The NIF link selection for which 2 HIF13-16 HIF13-16
links to use is random Slot 5
HIF17-20 HIF17-20
• Connections on the the other fabric
Slot 6
remains the same HIF21-24 HIF21-24
Slot 7
HIF25-28 HIF25-28
Slot 8
HIF29-32 HIF29-32
Increased Bandwidth Access to Blades
4 links, Discrète - Today 8 links, Discrète Up to 8 links, Port-channel
slot 1 slot 1
slot 2 F Fabric slot 2 F Fabric F Fabric
slot 3
slot 3
E Interconnect slot 4 E Interconnect E
slot 4
slot 5 Interconnect
slot 5
slot 6 X slot 6 X X
slot 7 slot 7
slot 8 slot 8
• Available bandwidth per • Available bandwidth per • Available bandwidth per

blade – 10Gb blade – 20Gb blade – up to 160Gb
• Statically pinned to • Statically pinned to • Statically pinned to Port-
individual fabric links individual fabric links channel
• Deterministic Path • Deterministic Path • Increased and shared
bandwidth
• Guaranteed 10Gb to
each blade • Higher Availability
Port-Channel Pinning
• No slot based pinning
• No invalid link count for NIF ports
VIC1200
adaptor with IOM
DCE links in
Port-Channel
Pinned
to Po
Gen-1 adaptor
with single 10G
link
UCS FI and IOM connectivity
Fabric Interconnect VIF calculation (6200 only)
1 2 3 4 5 6
 Every 8 10GbE ports (on FI) are controlled by the same Unified Port Controller (UPC)
 Connect fabric links from IOM to the FI to the same UPC
 For fabric port-channeling, Virtual Interface (VIF) namespace varies, depending on number and
how the fabric links are connected to the FI ports.
– Connecting to the same UPC (a set of eight ports), Cisco UCS Manager maximizes the number of
VIFs used in service profiles deployed on the servers.
– If uplink connections are distributed across UPC, the VIF count is decreased. For example, if you
connect seven (IOM) fabric links to (FI) ports 1-7, but the eighth fabric link to FI port 9, the number
of available VIFs is based on 1 link – IOM port 8 to FI port 9.
UCS FI and IOM connectivity
Fabric Interconnect VIF calculation cont’d (6200 only)
UPC
FI-A UPC UPC UPC UPC UPC UPC UPC
1 2 3 6 1 2 3 6
UCS 2208XP
UCS 2208XP UCS 2208XP
UCS 2208XP
1
1 1
1
2
2 2
2
3
3 3
3
4
4 4
4
5
5 5
5
6
6 6
6
7
7 7
7
8
8 8
8
IOM-B IOM-A IOM-B

IOM-A
2208 XP 2208 XP 2208 XP
2208 XP
 Recommended  Not recommended
 Maximize number of available VIFs to the  Minimal number of VIFs to the host
host
Virtual Interfaces (VN-TAG)
How Do Servers Communicate?
• We know servers with one mezz card present two (M81KR and non-Cisco
adapters) or 2 x 4 x 10G Base-KR ‘external’ or ‘northbound’ interfaces
• The OS knows nothing of this
• The OS sees PCI devices on the bus and loads device drivers for those devices
• In UCS, the Service Profile controls what the OS sees
• E.g.: a blade can be shown 6 x 10GE NICs and 2 x HBAs while another sees 8 x 10GE
NICs and no HBAs
• This means the northbound physical interfaces between the adapter and the
IOM can carry both Ethernet and FC traffic for several vNICs. We need a
mechanism to identify the origin server
•  Concept of Virtual Interface or VIF (see next slide)
Cisco UCS: Infrastructure Virtualization
Switchport Virtualization (vEth, vFC)
Fabric Interconnect
Cable Virtualization (VNTag)
vFC vEth vFC vEth
1 1 2 2 DCB Ethernet
Eth 1/1 Eth 1/2

Individual
Ethernets
Individual
Storage
(iSCSI, NFS, FC)
Blade or Rack
Service Profile PCIe Adapter
• # Adapters CPU
• Identity (MAC / WWN) MEM
• Firmware
• Settings I/O PCIe
Server Abstraction Adapter Virtualization (NIV)

Abstracting the Logical Architecture
Physical Logical
Fabric Fabric Fabric
Switch vFC vEth vFC vEth

1 1 1 1
 Dynamic, Rapid
Eth 1/1 Provisioning
 State abstraction
IOM A IOM A
 Location
Cable Independence
10GE 10GE
A
A  Blade or Rack
Adapter Physical Cable

vHBA vNIC
vHBA vNIC
1 1 Virtual Cable
1 1
(VN-Tag)
Service Profile
Blade (Server) (Server)
VN-Tag: Instantiation of Virtual Interfaces
• Virtual interfaces (VIFs) help distinguish between FC and Eth interfaces
• They also identify the origin server
• VIFs are instantiated on the FI and correspond to frame-level tags assigned to
blade mezz cards
• A 6-byte tag (VN-Tag) is preprended by Palo and Menlo as traffic leaves the
server to identify the interface
• VN-Tag associates frames to a VIF
• VIFs are ‘spawned off’ the server’s EthX/Y/Z interfaces (examples follow)
VN-Tag at the Adapter (Mezz Card) Level
• Connect to a server’s adapter and use “show-vifs”
VIFs
• Ethernet and FC are muxed on the same physical links  concept of virtual
interfaces (vifs) to split Eth and FC
• Two types of VIFs: veth and vfc
• Veth for Ethernet and FCoE; vfc for FC traffic
• Each EthX/Y/Z or Po interface typically has multiple vifs attached to it to carry
traffic to and from a server
• To find all vifs associated with a EthX/Y/Z or Po interface, do this:
Another Way to Find VIFs:
Server Connectivity
I/O for all Platforms
• Converged Network Adapter (CNA)
• Multi-form factor support
• mLOM* and PCIe based for rack servers
• mLOM* and Mezzanine for blade servers
mLOM (rack) PCIe
• Multi-speed suppot
• 10G, 20G, and 40G connectivity support
• Multi-management support
• CIMC – rack servers in standalone mode
• UCSM – rack and blade servers that are
connected to the FI either directly or IOM/FEX Mezzanine mLOM (blade)
* Note mLOMs are not interchangeable between rack and blade server
VIC Generation Comparison
Features VIC 1200 VIC 1300
PCIe Gen-2 x16 Gen-3 x16 (dual x8)
Speed 10G, 20G*, or 40G* 10G, 20G*, or 40G**
PPS x x
VIF 256 256+
Offloads IPv4 and IPv6 L3/L4 Checksum, TSO, LSO IPv4 and IPv6 L3/L4 Checksum, TSO, LSO
FCOE
QOS
Netflow
VM-FEX Technology (UCSM
Only - SRIOV)
Kernel Bypass Technology usNIC/Intel OpenMPI/DPDK usNIC/Intel OpenMPI/DPDK
Netqueue/VMQ
Network Overlay Offload NVGRE/VxLAN
ROCE Support v2
* Multiple 10G bundled in a port-channel

** Operate either 10G bundle or native 40G interface
VIC1x40 plus Port Expander card
 VIC 1200 and VIC1300
 PCIe Devices 256 (vNICs or vHBA)
 Support VM-FEX for ESX, RHEL-KVM and HyperV (SCVMM)
 Base option supports dual 2x10Gb or 40G options
 Option to enable all port of ASIC (Sereno)

 Fits in the Mezzanine slot of B200M3
 Port Expander has no PCIe presence
 It is a “passive connector” device
Connectivity IOM to Adapter
Up to 32 Gbps throughput per vNIC using flow based port-channel hash
2208 IOM 2208 IOM
 Implicit Port-channel between UCS 1240/VIC1340 or

1280/1380 VIC adapter and UCS 220x IOM or 2304
Side A Side B  7-Tuple Flow based hash
UCS VIC  A vNIC is active on side A or B.
vNIC1  A vNIC have access to up to 32 Gbps throughput .

VM Flows
VM
1. 10 Gb FTP traffic
2. 10 Gb UDP traffic
Block Diagram: UCS Fabric 62xx Details
MLOM only
UCS 6248 UCS 6248
Fabric
Interconnects 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module
IO Modules
2208XP 2208XP
Midplane
Adapter  Dual 2x10 Gbps port-channel from VIC 1240 to 2208

1x40 VIC Empty IO Modules
x16 Gen 3 x16 Gen 3
 No user configuration required
 vNIC flows are 7-tuple Load Balanced across links
Server Blade CPU 0 CPU 1
QPI Link
 Each individual flow limited to 10Gb
 Fabric Failover available
B200M4
UCS Blade Chassis
MLOM plus Port Expander
UCS 6248 UCS 6248
Interconnects
IO Modules
2208XP 2208XP
Port Channel 1
Midplane Port Channel 2
 Port Expander Passive

Adapter  Increase BW to 80Gbps
1240 VIC Port Exp
 Dual 4x10Gbps Port-channel
x16 Gen 2 x16 Gen 2  No user configuration required
CPU 0 CPU 1  vNIC flows are 7-tuple Load Balanced across links
Server Blade
QPI Link  Each individual flow limited to 10Gb
B200M3
UCS Blade Chassis
MLOM plus VIC1280
UCS 6248 UCS 6248
Interconnects
IO Modules
2208XP 2208XP
Midplane
 Adapter Redundancy
Adapter 1240 VIC
 Split vNIC across adapters
VIC1280
 4 2x10 Gbps Port-channel
x16 Gen 2 x16 Gen 2
 No user configuration required
CPU 0 CPU 1  vNIC flows are 7-tuple Load Balanced across links
Server Blade QPI Link  Each individual flow limited to 10Gb
B200M3
UCS Blade Chassis
Block Diagram: Next Gen UCS Fabric Details – B420M3
MLOM plus Port Expander plus VIC1280
UCS 6248 UCS 6248
Interconnects
IO Modules
2208XP 2208XP
Port Channel 1 Port Channel 2
Midplane
 Second I/O Adapter only support Port

4x10 4x10
Adapter
1240 VIC Port Exp VIC1280 Expander Card
x16 Gen 2 x16 Gen 2 x16 Gen 2
 Total BW is160 Gbps
 1240+Port Expander is a dual 4x10Gbps
Server Blade
CPU CPU CPU port-channel
QPI Link QPI Link  1280 is dual 4x10Gbps port-channel
 Non-configurable, hashing, FF
B420M3
UCS Blade Chassis
Topology Designs For Maximum Bandwidth
UCS 6248UP UCS 6248UP UCS 6248UP UCS 6248UP
UCS 2104 IOM UCS 2208 IOM UCS 2208 IOM UCS 2208 IOM
Side A Side A Side B Side A Side \B Side A Side B

Side B
1280 VIC or M81KR M81KR UCS 1280 VIC UCS 1280 VIC
• Shared IOM uplink • Shared IOM uplink • Dedicated IOM uplink • Shared IOM uplink
bandwidth of 10Gbps bandwidth of 80Gbps bandwidth of 10Gbps bandwidth of 80Gbps
• vNIC Burst up to 10Gbps • vNIC Burst up to 10Gb • vNIC Burst up to 10Gbps • vNIC Burst up to 32Gbps
• Shared IOM Uplink with 1 • Shared IOM Port-Channel *(IOM uplink limitation) *(PCIe Gen 2 limitation)
server with 8 servers • Dedicated IOM Uplink • Shared IOM Port-Channel
• Host port pinned to a • Host port pinned to a • Host port-channel pinned with 8 servers
discrete IOM uplink discrete IOM port-channel to discrete IOM uplink • Host port-channel pinned
to the IOM port-channel
IOM 2304 and Adapter Connection
VIC1240 and VIC1240 plus Port Expander
• Same connection as • Same connectivity as

the 2208XP IOMs the 2208XP IOMs
• Dual 20G via 2x10G in • Dual 40G via 4x10G
a port-channel in a port-channel
connection connection
• Port-channel is • Port-channel is
transparent to the OS transparent to the OS
• OS interface is 20G • OS interface is 40G
• Traffic is hashed • Traffic is hashed

across the 20G port- across the 40G port-
channel interface channel interface
VIC1340
• Dual 2x20G via 2x10G port-

channel connection
• Port-channel is transparent to
the OS
• OS interface is 20G
• Traffic is hashed across the
20G port-channel interface
VIC1340 plus Port Expander
• Dual native 40G interface

• Uses all 4 lanes
• OS sees 40G interface
• Packet is bit hashed across

the 4 lanes
Native 40G HIF
One Adapter or Two Adapter?
VIC1300 with FI-6332 and 2304 IOM
IOM-A IOM-B IOM-A IOM-B IOM-A IOM-B
PE
40Gbps UIF0 UIF1 40Gbps UIF0 UIF1 UIF0 UIF1
VIC1340 VIC1340 VIC1380
• Dual-40Gbps • Each adapter is a Dual-20Gbps

• Native 40G interface via 40G protocol • Adapter resiliency
• Better 40G efficiency because bit • Four- 2x10G port-channel
spray across 4 lanes • Hash based on traffic type
• Max BW for a given flow is 40G • Max BW for a given flow is 10G
• Total BW is 80G • Total BW to server is 80G
Fabric Forwarding Mode of
Operations
Fabric Forwarding Mode of Operations
LAN
• Switch mode: User configurable • End-host mode (EHM): Default mode

• Fabric Interconnects behave like • No spanning-tree protocol (STP)
• Active/Active for all links and VLANs
regular ethernet switches
• Port definitions
• STP parameters are lock
• Policy based forwarding
• VLAN/Mac based forwarding • No unknown unicast forwarding
End Host Mode
LAN
Spanning • Completely transparent to the
Tree network
• Presents itself as a bunch of hosts to
the network
FI A MAC • No STP – simplifies upstream
Learning connectivity
vEth 3 vEth 1
Fabric A
VLAN 10 MAC • All uplinks ports are forwarding –
Learning
never blocked
L2
Switching
VNIC 0 VNIC 0
Server 2 Server 1
End Host Mode
Unicast Forwarding
LAN • MAC/VLAN plus policy based
Server 2
forwarding
Uplink Ports • Server pinned to uplink ports
Deja-Vu
RPF
• Policies to prevent packet looping
FI • déjà vu check
• RPF
• No uplink to uplink forwarding
VLAN 10
vEth 1 vEth 3
• No unknown unicast or multicast
• igmp-snooping can be disable on per-
VLAN basis
VNIC 0 VNIC 0
Server 2 Server 1
End Host Mode
Multicast Forwarding
LAN • Broadcast traffic for a VLAN is
B B
pinned on exactly one uplink port (or
Broadcast
Uplink Listener
port-channel) i.e., it is dropped when
Ports per VLAN received on other uplinks
FI • Server to server multicast traffic is
locally switched
• RPF and déjà vu check also applies
vEth 1 vEth 3
for multicast traffic
B
VNIC 0 VNIC 0
Server 2 Server 1
Switch Mode
Root LAN
• Fabric Interconnect behaves like a
normal L2 switch
• Rapid-STP+ to prevent loops
• STP parameters are not configurable
MAC
Learning • Server vNIC traffic follows STP
vEth 3 vEth 1 forwarding states
VLAN 10
• Use VPC to get around blocked ports
L2 • VTP is not supported
Switching
• MAC address learning on both
VNIC 0 VNIC 0 uplinks and server links
Server 2 Server 1
Uplink Pinning
End Host Mode - Dynamic Pinning
LAN • UCSM manages the vEth pinning to
the uplink
• UCSM will periodically vEth
distribution and redistribute the
FI A vEths across the uplinks
Pinning
vEth 2 vEth 3 vEth 1
VLAN 10 Switching
VNIC 0 VNIC 0 VNIC 0
Server 2 Server 3 Server 1

End Host Mode – Individual Uplinks
Dynamic Re-pinning of failed uplinks
FI-A
Sub-second re-pinning Pinning
vEth 3 vEth 1
VLAN 10 Switching
Fabric A
All uplinks forwarding for all VLANs L2

VNIC stays up
GARP aided upstream convergence Switching
vSwitch / N1K
No STP VNIC 0
Sub-second re-pinning MAC A ESX HOST 1
No server NIC disruption VM 1 VM 2
MAC B MAC C
VNIC 0 Server 2
End Host Mode – Port
Recommended: Channel
Port Channel Uplinks Uplinks No disruption
No GARPs
needed
FI-A
Sub-second convergence Pinning
vEth 3 vEth 1
VLAN 10 Switching
Fabric A
More Bandwidth per Uplink L2

Per flow uplink diversity NIC stays up
Switching
No Server NIC disruption VNIC 0
vSwitch / N1K
Fewer GARPs needed ESX HOST 1

MAC A
Faster bi-directional convergence
VM 1 VM 2
Fewer moving parts
MAC B MAC C
VNIC 0 Server 2
RECOMMENDED
End Host Mode – Static Pinning
LAN Administrator Pinning Definition
vEth Interfaces Uplink

vEth 1 Blue
vEth 2 Blue
FI A vEth 3 Purple
Pinning
vEth 2 vEth 3 vEth 1 • Administer controls the vEth pinning
VLAN 10
Switching • Deterministic traffic flow
• Pinning configuration is done under the
LAN tab -> LAN Pin groups and
assigned under the vNIC
VNIC 0 VNIC 0 VNIC 0 • No re-pinning with in the same FI
• Static and dynamic pinning can co-
Server 2 Server 3 Server 1 exist
Which uplink is the servers’ vEth pinned to?
TME-UCS6100-A(nxos)# sh pinning server-interfaces
---------------+-----------------+------------------------+------------
-----
SIF Interface Sticky Pinned Border Interface Pinned
Duration
---------------+-----------------+------------------------+------------
-----
Vlan1 No - -
sup-eth0 No - -
sup-eth1 No - -
sup-eth2 No Po2 3d 11:51:16
sup-eth3 No - -
Eth1/9 No - -
Eth1/10 No - -
Eth1/11 No - -
Eth1/12 No - -
Veth1093 No Po1 13d 47:39:22 TME-UCS6100-A(nxos)# sh pinning border-interfaces
Veth1094 No Po1 13d 47:39:22
Veth1099 No Po1 17:39:19 --------------------+---------+------------------------------
Veth1103 No Po2 17:37:19 Border Interface Status SIFs
Eth3/1/1 No - - --------------------+---------+------------------------------
Po1 Active Veth1093 Veth1094 Veth1099
Po2 Active sup-eth2 Veth1103
Eth1/6 Down
Eth1/7 Down
Eth1/8 Down
Total Interfaces : 5
Fabric Failover (FF)
Fabric Failover
End Host Mode (only)
• Fabric provides NIC failover capabilities
LAN SAN A SAN B
chosen when defining a service profile
• Traditionally done using NIC bonding
UCS Fabric driver in the OS
Interconnects
• Provides failover for both unicast and
multicast traffic
Chassis
Fabric Extender Fabric Extender • Works for any OS on
bare metal and hypervisors
UCS-6200-A /chassis/server/adapter/host-eth-if # show vif
Adapter Adapter VIF:
vNIC
vNIC
vNIC
vNIC
ID Fabric ID Transport Tag Status Oper State

---------- --------- --------- ----- ----------- ----------
CiMC CiMC 1201 A Ether 0 Allocated Active
1202 B Ether 0 Allocated Passive
Half Width Blade Half Width Blade

Fabric Failover
Bare Metal OS
1 2 1 2
Upstream Switch Upstream Switch

15 15
16 14 14 16
Uplink Ports 7 8 8 7 Uplink Ports
UCS FI-A UCS FI-B
VLAN 10 VLAN 20 VLAN 10 VLAN 20
HA Links
1 2 3 4 5 6 1 2 3 4 5 6
Server Ports Server Ports
1 2
Fabric Ports
3 4 UCS 1 2
Fabric Ports
3 4
FEX-A Blade FEX-B

Backplane 1
Chassis
2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
Backplane
Ports Ports
Blade Server
Eth 1/1/4 Adapter Eth 1/1/4
MAC –A Eth 0 MAC –B Eth 1
PCI Bus
Bare Metal Operating System
Windows / Linux
Fabric Failover
Bare Metal OS
1 2 1 2
Upstream Switch Upstream Switch

15 15
MAC-A gARP
16 14 14 16
Uplink Ports 7 8 8 7 Uplink Ports
UCS FI-A UCS FI-B
VLAN 10 VLAN 20 VLAN 10 VLAN 20
HA Links
1 2 3 4 5 6 1 2 3 4 5 6
Server Ports Server Ports
1 2
Fabric Ports
3 4 UCS 1 2
Fabric Ports
3 4
FEX-A Blade FEX-B

Backplane 1
Chassis
2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
Backplane
Ports Ports
Blade Server
Eth 1/1/4 Adapter Eth 1/1/4
vNIC stays UP
MAC –A Eth 0 MAC –B Eth 1
PCI Bus
Bare Metal Operating System
Windows / Linux
Topologies
Network Connectivity
• Not recommended • Active/Active • Recommended
• Double oversubscription • VIF pinning • Active/Active
• GARP on uplink failure
• Hashing on traffic
vPC/VSS
Nexus Nexus
FEX FEX
FI-A FI-B FI-A FI-B

FI-A FI-B
Layer 2 Disjoint Topology
• A vNIC can only participate in one
Production Backup
VLAN 10-20 VLAN 30-40 L2 network upstream
• Both dynamic and static pinning
methods are supported
Fabric Interconnect - A Fabric Interconnect - B
End Host Mode End Host Mode
IOM-A IOM-B
VLAN 11 VLAN 30
Adapter
vNIC-A
vNIC-B
Half Width Blade
VLAN Groups
• Object for a set of VLANs
• Not just for Disjoint L2
• Easier to manage and configure
• 3.1(3) vlan groups are now
configurable from vNIC and
vNIC templates
Management Overview
UCS Director
• vPC to the ACI leafs

SAN A SAN B
• Enable LLDP to the
vEth
UCSM /
UCS Central
Ethernet and
Ethernet
FCOE
FC FCOE
Recommendations
IPv6 Management Support
• UCS FI management can be configured with IPv6 address
• IPv6 capable external services (i.e., NTP, SSH, TACACs, HTTP/HTTPs, etc)
Inband Management for CIMC
• Separate server management (CIMC) traffic from UCSM
• Designating servers’ CIMC into different groups
• Higher bandwidth
• 10G vs 1G
Faster and Better: Link layer enhancements
• Uplink ports only
• Faster link failure detection with UDLD
• Layer 2 protocol that runs on top of the physical layer to help detect mis-wiring and uni-
directional communication
• UDLD Interval 7-90 seconds, with a detection time of 3x the interval
• LACP (pertain to uplinks only)

• Choices between different timers, slow (30 sec) or fast (1 sec). Default is slow.
Provides very fast failure detection
• Enable suspend-individual link. Default is disable
• Match settings of the LAN Switch
PVLAN Enhancements
• Promiscuous on Appliance Port
• Community Support
• PVLAN trunking on the vNIC (extend PVLAN to the virtual switches)
Netflow+
• Improve workload visibility
• Capacity planning
• Security
• Troubleshooting
Use Fabric Evacuation
Customer benefits
• Ensure smooth failover of network, storage and

management traffic while upgrading UCS
Infrastructure
• Minimal to no disruption to application traffic
during UCS FI upgrade
Feature details
• Evacuate all traffic flows through a Fabric
Interconnect from all servers attached to it via
IOM or FEX during system upgrade
• Similar to graceful disable of all uplinks
• Maintenance Mode for Fabric Interconnect
Normal Operational State
Uplinks
• Traffic is flowing on both
UCS FI-A UCS FI-B Fabric
Primary Sub-ordinate
• I/O is setup for some NIC
Fabric
Links teaming for HA
IOM-A IOM-B
HIF
vNIC vHBA vNIC vHBA

I/O A A B B
OS
UCS Blade Chassis
Invoke Fabric Evacuation
• HIF ports on the sub-
ordinate side are brought
Uplinks
down
UCS FI-A UCS FI-B
Primary Sub-ordinate • Corresponding vNICs and
Fabric vHBAs will be brought down
Links
• If HA is configured for the
IOM-A IOM-B
server, then traffic will
HIF
failover to the redundant
I/O vNIC
A
vHBA
A
vNIC
B
vHBA
B path
OS
UCS Blade Chassis
UCSM Firmware Auto Install Enhancements
Customer benefits
• Provides smoother transition of network failover during Fabric
Interconnect firmware upgrade.
Feature details
• Adding “Fabric Evacuation” option to the UCSM Infrastructure
Firmware Auto Install workflows
• Using Fabric Evacuation, network flows are forcefully

evacuated on a Fabric Interconnect, prior to that Fabric
Interconnect activating.
• Network Flow failover is no longer triggered by heartbeat
loss, and instead is forced more gracefully via
evacuation.
• More controlled failover of network flows during FI activation
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

Brkini 2025

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkini 2025

Uploaded by

Copyright:

Available Formats

UCS Networking –

Cisco Spark spaces will be cs.co/ciscolivebot#BRKINI-2025

Fabric Interconnect (FI)

UCS Fabric Extender (FEX)

Compute (Blade and Rack)

UCS Virtual Interface Card (VIC)

UCS 6324 (Mini) UCS 6300

UCS 6296 & 6248

UCS 6120 & 6140

SHIPPING EOL / EOS

FI 6200 Series FI 6300 Series

* Using 40G to 4x10G breakout cables ** Requires QSA module

IOM 2200 Series IOM 2300

• Requires UCSM 3.0

Native Fibre Channel Lossless Ethernet:

Base card – 32 Unified Ports GEM – 16 Unified Ports

 16 SFP+ Unified Ports  24 QSFP+ Ports

• 16 Unified Ports • 16 Unified Ports

 Breakout Port Creates (4) 10Gb Ports

Fabric Extenders Chassis 1 Chassis 20

2x 1 Link 2x 2 Link 2x 4 Link 2x 8 Links

Chassis Control ASIC Woodside Woodside

Chassis 2204 CoS 8 8

Fabric interface state:

Double the Fabric Uplinks

Midplane Quadruple the Downlinks

Adapter mLOM Mezz

UCS Blade Chassis

fex-1# sh platform software woodside rate

• Dual-Core Freescale CPU Memory

• HDMI Connector (Dongle) Mgmt

1 Internal backplane ports to blades 32

NIF2 Slot 3 NIF2

NIF2 Slot 3 NIF2

NIF2 Slot 3 NIF2

Fabric interface state:

on the OS, then traffic will failover NIF4 Slot 4

• Blades will redistribute the NIF2 Slot 3

• Available bandwidth per • Available bandwidth per • Available bandwidth per

IOM-B IOM-A IOM-B

Eth 1/1 Eth 1/2

Server Abstraction Adapter Virtualization (NIV)

Switch vFC vEth vFC vEth

Adapter Physical Cable

* Multiple 10G bundled in a port-channel

 Option to enable all port of ASIC (Sereno)

 Implicit Port-channel between UCS 1240/VIC1340 or

Side A Side B  7-Tuple Flow based hash

UCS VIC  A vNIC is active on side A or B.

vNIC1  A vNIC have access to up to 32 Gbps throughput .

Adapter  Dual 2x10 Gbps port-channel from VIC 1240 to 2208

 Port Expander Passive

 Second I/O Adapter only support Port

Side A Side A Side B Side A Side \B Side A Side B

• Same connection as • Same connectivity as

• Traffic is hashed • Traffic is hashed

• Dual 2x20G via 2x10G port-

• Dual native 40G interface

• Packet is bit hashed across

40Gbps UIF0 UIF1 40Gbps UIF0 UIF1 UIF0 UIF1

VIC1340 VIC1340 VIC1380

• Dual-40Gbps • Each adapter is a Dual-20Gbps

• Switch mode: User configurable • End-host mode (EHM): Default mode