Module 5: FCoE Theory of Operations

Module 5 – FCoE Theory of Operations

Upon successful completion of this module, you should be
able to:
– Describe the current FCoE Architecture
– Describe the different MAC addresses used by an FCoE ENode
– Describe the FCoE Initialization Protocol

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 1

After successfully completing this module, you should be able to:

 Describe the current FCoE Architecture
 Describe the different MAC addresses used by an FCoE ENode
 Describe the FCoE Initialization Protocol

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 1

Module 5: FCoE Theory of Operations

Lesson 1: FCoE Overview

Upon completion of this lesson, you will be able to:
 Describe the current FCoE architecture, and explain how
FCoE data travels through the various network layers.

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 2

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 2

 Module 5: FCoE Theory of Operations

Network Stack Comparison

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP: FC-4 FCP: FC-4 FCP: FC-4

FC FC FC – 3
Common Services

FCIP Less Overhead

than FCIP, iSCSI
TCP TCP

IP IP FCoE FC – 2
Frame/Flow Ctrl

Ethernet Ethernet Ethernet FC – 1

Encoding

PHYSICAL WIRE FC - 0
SCSI iSCSI FCIP FCoE FC
© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 3

FCoE uses FC-4 and FC-2 as the upper layer network protocols, and encapsulates the FC-2
frames into Ethernet for transport. The FCF Switch will strip off the Ethernet information and
forward the frame to the FC network (or a directly connected device? – is this supported at GA?)

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 3

 Module 5: FCoE Theory of Operations

The FCoE Frame

Word
0 Destination MAC Address

2 Source MAC Address

3 IEEE 802.1Q Tag (optional)

4 Ethertype = FCoE (8906hex) Ver Reserved

5 Reserved

6 Reserved

7 Reserved SOF

8
Encapsulated FC Frame  24 byte header + 0 to 2112 byte payload + 4 byte CRC

(n words)  7 words min; 560 words max

n+8 EOF Reserved

n+9 FCS

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 4

Reference: FIBRE CHANNELBACKBONE – 5 (FC-BB-5) REV 1.01

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 4

 Module 5: FCoE Theory of Operations

Data Path Through the FCoE Layers

F_Port F_Port E_Port
SCSI HBA Driver
FC-0 FC-0 FC-0

FC-4: SCSI-FCP FC-1 FC-1 FC-1

FC-2 FC-2 FC-2

FC-3

FC-2 FC Switching Element (FCF)

FC Entity (VN_Port) FC Entity (VF_Port) E_Port

FCoE_LEP FCoE_LEP FC-2

Ethernet MAC Ethernet MAC FC-1

Ethernet Physical Ethernet Physical FC-0

10 GigE 10 GigE
Ethernet Cable Fibre Channel Cable
Port Port

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 5

The FCoE Logical End Point is a virtual Fibre Channel Entity. FCoE_LEP = FCoE Link
Endpoint
Ethernet cable is part of Lossless Ethernet network
The FCoE_LEP is the functional entity performing the encapsulation of FC frames into FCoE
frames in transmission and the de-encapsulation of FCoE frames into FC frames in reception.
VN_Ports are created through a successful FLOGI or FDISC operation.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 5

 Module 5: FCoE Theory of Operations

ENode (CNA) – Simplified Model

CNA

FC Node
FCoE_LEP … FCoE_LEP

10 GE … 10 GE
port port

One MAC address per

FCoE Link End Point

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 6

An ENode is an Ethernet NIC that can operate as a Fibre Channel N_Port.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 6

 Module 5: FCoE Theory of Operations

ENode Implementation (CNA)

FC

SERDES
10 GE Menlo
ASIC
10 GE 10 GE

PCIe Bus

 Off-the-shelf NIC and HBA ASICs from QLogic and Emulex

 Dual 10 Gigabit Ethernet ports supporting FCoE
 Support for native drivers and utilities:
– Customer-certified stacks

 Replaces multiple adapters per server

 Consolidates 10 Gigabit Ethernet and Fibre Channel on single interface
 Minimum disruption in existing customer environments
© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 7

The Menlo ASIC was developed by Nuova Systems Inc. Nuova has since been acquired by
Cisco Systems.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 7

 Module 5: FCoE Theory of Operations

Functional Model of FCoE Switch

One MAC address per

FCoE Link End Point FC
port
FC
FCF (Switch) port
FCoE_LEP FC
port
FC
Ethernet Switch port

10 GE 10 GE 10 GE 10 GE 10 GE 10 GE 10 GE 10 GE
port port port port port port port port

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 8

FCF = Fibre Channel Forwarder

LEP = Link End Point

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 8

 Module 5: FCoE Theory of Operations

Lesson 2: Addressing
Upon completion of this lesson, you will be able to:
 Explain how Addresses are assigned for FCoE nodes

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 10

Note: FIP (Fabric Initialization Protocol) is not supported with this version of code on the
Connectrix NEX-5020 switch. When FIP is supported, it will be added to this lesson. In the
meantime, students can refer to the description of FIP in the FC-BB-5 standards document.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 10

 Module 5: FCoE Theory of Operations

Addressing
Fibre Channel
SAN

FCID =01xxyy
MAC = A MAC = B
Domain ID = 1
FCID = 05xxyy Domain ID = 5
FC Switch
NEX-5020
CNA
Host
Storage

Dest = MAC B
Source = MAC A
D_ID = 01xxyy D_ID = 01xxyy D_ID = 01xxyy
S_ID = 05xxyy S_ID = 05xxyy S_ID = 05xxyy
FC Frame FC Frame
Ethernet Frame

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 11

This example shows how both MAC addresses and Fibre Channel addresses would show up in a
frame being forwarded from a CNA to a storage port.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 11

 Module 5: FCoE Theory of Operations

Option 1: Server Provided MAC Addresses (SPMA)

 Adapter uses burned-in or configured MAC address:
– Consistent with the Ethernet model
 FCF needs a table to map between MAC addresses and FC_IDs

MAC
Burned in or Configured
Address
48 bits

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 12

The MAC addresses on the NEX-5020 switches associated with the VE_Ports and VF_Ports are
universal MAC addresses from a switch pool assigned to the manufacturer by IEEE.
VN_Port addresses can be assigned to the ENode (CNA) in one of two ways. This slide shows
option 1 – where the MAC address would be burned in by the CNA manufacturer, or configured
by an administrator. This is called SPMA or “Server Provided MAC Address”. This option is
not used at this time.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 12

 Module 5: FCoE Theory of Operations

Option 2: Fabric Provided MAC Addresses (FPMA)

 MAC address assigned for each FC_ID:
– Consistent with the Fibre Channel model
 Uses OUIs with U/L = 1 (Local addressing), called FC-MAPs
 Multiple FC-MAPs may be supported (one per SAN)
 No table needed for Encapsulation
* Connectrix NEX-5020 uses FPMA *
24 bits 24 bits
FC-MAP FC-ID
(0E-FC-00) 7.8.9

MAC FC-MAP FC-ID

Address (0E-FC-00) 7.8.9
48 bits
© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 13

The Connectrix NEX-5020 uses option 2: “Fabric Provided MAC Addresses” or FPMA. With
this option, the switch assigns the MAC address to the attached VN_Port (CNA). The VN_Port
is created and the MAC address is assigned during the Fibre Channel Login process.
FPMA addresses are not universal, but instead are local addresses (local to the SAN). These
addresses have OUIs with the U/L bit set to 1 (Local).
 FC-MAP = Fibre Channel MAC Address Prefix
 The FPMA address is constructed by adding a 24-bit FC-MAP prefix to a 24-bit FCID.
 With FPMA, each FCID is assigned its own MAC address.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 13

 Module 5: FCoE Theory of Operations

Planning & Design Considerations

 Use Connectrix NEX-5020 switch as a data center
access layer switch enabling I/O consolidation using
FCoE
 Use minimum of two CNAs per host connecting to
separate Connectrix NEX-5020 switches for high
availability
 Use multiple ISLs with port channels for connecting to
existing fabric
– Standard best practices for MDS performance and high availability

 Check Powerlink for the specifications of CNAs qualified

with servers and operating systems

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 14

Shown are some of the planning and design considerations. Please take a moment to review
them. The next generation of data center using FCoE with 10 Gbps enhanced Ethernet is
expected to evolved in a phased approach. Before implementing the new technology in a
production environment, careful evaluation needs to be done.
When designing the solution, it is recommended to follow the best practices for traditional SAN
and Ethernet designs. For additional details, Please see EMC Fibre Channel over Ethernet
(FCoE) Installation and Configuration Guide, P/N: 300-008-103-A01.

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 14

 Module 5: FCoE Theory of Operations

Module Summary
Key points covered in this module:
– An ENode is an Ethernet NIC that can operate as a FC VN_Port.
– The Connectrix NEX-5020 is an FCoE Forwarder (FCF).
– Current FCoE standards support FCoE between an ENode and a
directly attached FCF deployed as an access layer switch.

© 2008 EMC Corporation. All rights reserved. Module 5: FCoE Theory of Operations - 16

These are the key points covered in this module. Please take a moment to review them

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 16