What’s NOT FC-BB-5

 FC-BB-5 doesn’t deal with how lossless is realized in

Ethernet
no Priority Flow Control, Bandwidth Management, etc.

 FC-BB-5 doesn’t deal with management functions

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1
 IEEE DCB standards status
DCB technologies allow Ethernet to be lossless and to
manage bandwidth allocation of SAN and LAN flows
Feature / Standard Standards Status
IEEE 802.1Qbb
PAR approved
Priority Flow Control (PFC)
Enable multiple traffic types to share a common 1.0 published
Ethernet link without interfering with each other

IEEE 802.1Qaz
PAR approved
Bandwidth Management (ETS)
Enable consistent management of QoS at the 1.0 published
network level by providing consistent scheduling

Data Center Bridging Exchange

Protocol (DCBX)
This is part of IEEE 802.1Qaz
Management protocol for enhanced Ethernet
capabilities

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2
Data Center Ethernet:
PFC & Bandwidth Management

CoS based
Priority Flow Control Bandwidth Management
Transmit Queues Receive Buffers
Ethernet Link Offered Traffic 10 GE Realized Traffic Utilization
Zero Zero
Zero
3G/s 3G/s 2G/s 3G/s HPC Traffic 2G/s
One One 3G/s
Two Two
3G/s Storage Traffic 3G/s
Three STOP PAUSE Three Eight 3G/s 3G/s 3G/s 3G/s
Virtual
Four Four
Four Lanes
Five Five 3G/s LAN Traffic 5G/s
3G/s 4G/s 6G/s
Six Six
Six 4G/s

Seven
Seven Seven
t1 t2 t3 t1 t2 t3

• Enables lossless behavior • Enables Intelligent sharing of

for each class of service bandwidth between traffic classes
• PAUSE sent per virtual lane control of bandwidth
when buffers limit exceeded • 802.1Qaz Enhanced Transmission

05/25/20
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved.
Nuova Systems Inc.
Cisco Confidential
3 3
DCBX Overview
Auto-negotiation of capability and configuration
Priority Flow Control capability and associated CoS values

Allows one link peer to push config to other link peer

Link partners can choose supported features and willingness to accept

Discovers FCoE Capabilities

Responsible for Logical Link Up/Down signaling of
Ethernet and FC
DCBX negotiation failures will result in:
vfc not coming up
Per-priority-pause not enabled on CoS values with PFC configuration

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf
http://www.ieee802.org/1/files/public/docs2008/
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4
 FCoE control plane

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5
 FIP: FCoE Initialization Protocol
 FCoE VLAN discovery
Automatic discovery of FCoE VLANs
 Device discovery
ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual
Links
VE_Port capable FCF-MACs discover other VE_Port capable FCF-MACs for
VE_Port to VE_Port Virtual Links
The protocol verifies the Lossless Ethernet network supports the required Max
FCoE Size
 Virtual Link instantiation
Builds on the existing Fibre Channel Login process, adding the Negotiation of the
MAC address to use
Fabric Provided MAC Address (FPMA), or
Server Provided MAC Address (SPMA)
 Virtual Links maintenance
Timer based
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6
Server Provided MAC Fabric Provided MAC
Addresses Addresses
Adapter uses burned-in or configured MAC address assigned for each FC_ID:
MAC address: Consistent with the Fibre Channel model
Multiple FC-MAPs may be supported
Consistent with the Ethernet
model One per SAN
No table needed for Encapsulation
FCF needs a table to map between
MAC addresses and FC_IDs Multiple MACs may be needed for NPIV

FC-MAP FC-ID
MAC (0E-FC-00) 7.8.9
Address 24 24
bits bits
Burned in or Configured FC-MAP FC-ID
(0E-FC-00) 7.8.9
48 48
bits bits
Cisco Nexus 5000 uses FPMA
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7
 Initial Login Flow ladder
ENode FCoE Switch

VLAN VLAN
Discovery Discovery

Solicitat
ion FIP:
FCF FCF FCoE
Discovery Advertisement Discovery Initialization
Protocol

FLOGI/FDISC FLOGI/FDISC Accept

FC Command FCOE
FC Command responses Protocol

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8
FCoE data plane

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 17
 ENode: Simplified Model
 ENode (FCoE Node): a Fibre Channel HBA implemented within an
Ethernet NIC aka CNA (Converged Network Adapter)
 FCoE LEP : The data forwarding component that handles FC
frame encapsulation/decapsulation
 FCoE Controller is the functional entity that performs the FIP and
instantiates VN_Port/FCoE_LEP pairs.

FC Node
FCoE_Controller FCoE_Controller

FCoE_LEP FCoE_LEP

Enet Enet
port port
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18
 FCoE Switch: Simplified Model

 FCF (Fibre Channel Forwarder), the forwarding entity

inside an FCoE switch

FC
port
FCoE Switch
FC
FCF port
FCoE_Controller
FC
FCoE_LEP
port

Ethernet Bridge FC
port

Eth Eth Eth Eth Eth Eth Eth Eth

port port port port port port port port

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 19
FCoE Network
Topology

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 20
 FCoE: Initial Deployment
SAN A SAN B
10GE
Backbone

VF_Ports Nexus 5000 (FCF)

VN_Ports
10GE

4/8 Gbps FC

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21
 FCoE: Adding Blade Servers
SAN A SAN B
10GE
Backbone

VF_Ports

10GE
VN_Ports 4/8 Gbps FC

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 22
 FCoE: Adding Native FCoE Storage
SAN A SAN B
10GE
Backbone

VN_Ports

VF_Ports

10GE
VN_Ports 4/8 Gbps FC

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23
 FCoE: Adding VE_ports
SAN A
SAN B
10GE
Backbone

VE_Ports

VF_Ports

10GE
VN_Ports 4/8 Gbps FC

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24