Fibre Channel Over Ethernet

Fibre Channel over Ethernet (FCoE)
John L Hufferd, Brocade

2008 Storage Networking Industry Association. All Rights Reserved.
2
SNIA Legal Notice
The material contained in this tutorial is
copyrighted by the SNIA.
Member companies and individuals may use this
material in presentations and literature under the
following conditions:
Any slide or slides used must be reproduced without
modification
The SNIA must be acknowledged as source of any
material used in the body of any document containing
material from these presentations.
This presentation is a project of the SNIA
Education Committee.
3
Abstract
A new concept is currently moving through the
Fibre Channel (T11) standards committee called
Fire Channel over Ethernet (FCoE).
The FCoE standard will specify the
encapsulation of Fibre Channel frames into
Ethernet Frames and the amalgamation of these
technologies into a network fabric that can
support Fibre Channel protocols and other
protocols such as TCP/IP, UDP/IP etc.
The tutorial will show the Fundamentals of the
FCoE concept and describe how it might be
exploited in a Data Center environment.
4
Agenda
Introduction
Goals & Requirements
Consolidation
Architecture
Discovery & Link Instantiation
Topologies
Scenarios
Summary
5
Introduction
This presentation provides an overview of Fibre
Channel over Ethernet (FCoE)
One should think about FCoE as placing the FC
protocol on a new physical link
Ethernet links instead of physical FC links
But it is still Fibre Channel
The protocol is being defined in the INCITS Fibre
Channel (T11) technical committee
Many details of the protocol still need to be defined
but the significant major issues have been resolved
Target T11 Standards completion is the 2H08
6
The Origins of Data Center Ethernet
The technology has evolved continuously, showing a great ability to
adapt to new technologies and increasing business requirements
Increasing Scalability, Feature, Function
I
n
c
r
e
a
s
i
n
g

P
e
r
f
o
r
m
a
n
c
e
Integration of Carrier-grade features
Incremental Protocol Enhancements
Logical Partitioning
Evolution from shared media to dedicated media
1973
2008-2009 CEE
Introduction of Ethernet
10Mbps
100Mbps
1Gbps
10Gbps
(Converged Enhanced Ethernet)
7
Goals/Requirements
8
FCoE Goals/Requirements (part 1)
FCoE Fabrics must be built with FCoE Switches
Switches with Ethernet ports that provide FCoE capabilities and
services Called, in the standard, an FC Forwarder FCF
Switches that include the functions of traditional FC switches
Standard Ethernet switches may also exist in the fabric but switches
with FCoE capabilities are required
FCoE fabrics must operate seamlessly with real FC Fabrics
FC services must operate identically on FCoE fabrics and
Fibre Channel fabrics
FCoE must support all Fibre Channel advanced features
(e.g. virtual fabrics, IFR, security, etc.) transparently
FCoE is NOT a replacement for FCIP or iFCP
FCIP & iFCP use TCP/IP
FCIP/iFCP is for inter-switch links beyond the Data Center
9
Integrated Ethernet & FCoE switch (FCF)
with FC connections
Some implementations may combine the features and capabilities of an Ethernet
Switch with the features and capabilities of a FC switch
Support Ethernet and IP standards for switching, pathing and routing
Support FC standards for switching, pathing and routing
Support current and enhanced Ethernet Standards
Adapt between FCoE and FC
An FCoE Port (N_Port, F_Port or E_Port)
Has the same function as in FC
But is layered on top of Ethernet
Called VN_Port, VF_Port and VE_Port
Because many logical (virtual)
Ports can share one physical port
Note:
FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol
Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as
STP (Spanning Tree Protocol) and MSTP (Multiple STP)
See additional FSPF and STP explanation in Appendix
FC FC FC
CEE Ethernet Ports
(with IP & FCoE VF_Port &
VE_Port capabilities)
10
Connections to a Combo FCoE Switch
Applications
TCP UDP
IP
SCSI
Fibre Channel
FCoE
Lossless Ethernet MAC (CEE)
Fibre Channel is carried over lossless Ethernet as a L3 protocol
Combo Lossless Ethernet (CEE) Switch
with FCoE Switch (FCF) capabilities
(FCoE VN_Port)
Ethernet port with IP & FCoE VF_Port
capabilities
IP address 123.45.67.89
iSCSI
SCSI
11
FCoE requires specific Ethernet extensions to be implemented
Lossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE)
configurations are required
J umboframe support is required (not a standard, but widely available)
Deployments of FCoE should utilize the advances in Ethernet
currently being discussed in IEEE 802.1, specifically:
Priority-based Flow Control (PFC)
Enhanced Transmission Selection (ETS)
DCB Capability eXchange Protocol (DCB CXP)
These 802.1 advances are important for Consolidated
Flows (Messaging, Clustering and Storage)
This set of functions has been called Data Center Ethernet, or CEE
Converged Enhanced Ethernet (intended for a Data Center
Environment)
FCoE should require no changes to FC software
12
Consolidation
13
Dramatic reduction in adapter, switch ports and cabling
4-6 cables to 2 cables per server
Seamless connection to the installed based of existing SANs
and LANs
Requires high performance lightweight frame mappers vs.
heavy weight gateways
FCoE has no need to terminate and re-initiate a SCSI connection
(e.g. iSCSI to FC)
Effective sharing of high bandwidth links
High End 10GE Server & NIC/HBA
Consolidation
Today
With CEE
OS3
DB Server
OS2
App Server
OS1
Web Server
Messaging
MPI
RDMA
FC
HBA
OS3
DB Server
OS2
App Server
OS1
Web Server
IB/Ethernet
Cluster
Hyper Visor
(vmWare, Xen, etc)
E-HBA
(CEE)
Data Center
CEE Network
Ethernet
TCP/IP
FC SAN
Data Center
CEE Network
E-HBA
(CEE)
E-HBA
(CEE)
NIC
TCP acceleration.
MPI, RDMA
over Ethernet
FCoE
OS3
DB Server
OS2
App Server
OS1
Web Server
14
Architecture
15
FC Encapsulation Into Network Packets
(2 FCoE Related Packet types)
IEEE 802.3
Layers
FC Levels
(Unchanged)
FCoE Mapping
FC-4
FC-3
FC-2
FC-1
FC-0
FC-4
FC-3
FC-2
MAC
PHY
Frame Check Sequence
(CRC)
Protocol control information: Version, SOF, EOF, etc.
FC Imbedded Frames: Same as in Physical FC
Ethernet
Header
FCS
FCoE
Header
FC Header
SCSI Commands/Data
Ethertype
FCoE
Fibre Channel over Ethernet ( Fibre Channel over Ethernet (FCoE FCoE) Packets ) Packets
Protocol control information: Version, Op-codes, etc.
Discovery and Login/Logout Parameters
Ethernet
Header
FCS
FIP
Header
Descriptors
Ethertype
FIP
FCoE Initialization Protocol ( FCoE Initialization Protocol (FIP FIP) Packets ) Packets
Ethernet Header provides things needed for the
physical network, including Ethertype
16
FIP Protocol and FCoE Protocol
Discovery Phase
FCFs Discovery each other and form a Fabric
ENodes and FCFs Discover
Potential VN_Port VF_Port pairing
Capabilities of Potential pairing
Login Phase
ENodes chose among discovered FCFs Ports
Creates association between ENode Ports and FCF Ports
VN_Port VF_Port Logical FC Link
Two allowed alternatives for the ENode MAC Addressing
Fabric Provided MAC Addresses (FPMA)
Server Provided MAC Addresses (SPMA)
Chosen by FCF (FPMA & SPMA Described latter)
Uses: FLOGI, FLOGI ACC, LOGO,
Data Transfer Phase
PLOGI/PRLI
All other FC protocol frames (ELS, FC4 ULPs. etc.)
FCoE
Initialization
Protocol (FIP)
FCoE Protocol
17
FIP Operation Format
Word 31-24 23-16 15-8 7-0
0 Destination MAC Address (6 Bytes)
1
2 Source MAC Address (6 Bytes)
3 ET=FIP (16 bits)
Ver (4b)
Reserved (12 bits)
4 FIP Operation Code
5
Reserved FIP subcode Descriptor List Length
6
Flags
F
P
S
P
S
F
PAD to minimum length or mini-Jumbo length
n
n+1 Ethernet FCS
Optional IEEE 802.1q
4 Byte
Tag goes here
Descriptor list
varies
In size
Ethernet frame
size
Is 64Bytes to 2220Bytes
FIP Operation Code Reserved FIP SubCode
Descriptor List Length Flags
FP SP S F
Descriptor List
Solicited bit FCF bit Capability Bits (SPMA or FPMA)
See Appendix for
Descriptor list items
18
FCs Encapsulation in Ethernet (FCoE)
Word 31-24 23-16 15-8 7-0
0 Destination MAC Address (6 Bytes)
1
2 Source MAC Address (6 Bytes)
3 ET=FCoE (16 bits)
Ver (4b)
Reserved (12 bits)
4 Reserved
5 Reserved
6 Reserved SOF (8 bits)
7 Encapsulated FC Frame
FC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)
(including FC-CRC)
n
n+1 EOF (8 bits) Reserved
n+2 Ethernet FCS
Optional IEEE 802.1q
4 Byte
Tag goes here
This field varies
In size
Ethernet frame
size
Is 64Bytes to 2220Bytes
19
FCF Model
Link
End
Point
(LEP)
20
ENode (HBA) Model
Each ENode (HBA) may have multiple Physical Ethernet Ports
Each Physical Port may have multiple Logical VN_Ports
Link
End
Point
(LEP)
21
Model of the ENODE with Multiple
Logical FC interfaces
FCoE
Controller
Lossless Ethernet MAC
Ethernet_Port
FC
Entity
VN_Port
FCoE
Entity
FCoE_LEP
FC-3 /FC-4s
MAC Address
of Burnt-in
MAC
FC
Entity
VN_Port
FCoE
Entity
FCoE_LEP
FC-3 /FC-4s
. . .
For each logical N_Port
(VN_Port) there is one FLOGI
and perhaps 100s of FDISC
Each VN_Port is seen by
the Host as a separate
(logical) FC connection
The number of (logical) FC
connections is
implementation dependent
Only one MAC Address is required for the FCoE Controller and the VN_Ports on a
single physical MAC (aka Server Provided MAC Address SPMA)
FCF may chose to specify new MAC addresses for each VN_Port (aka Fabric
Provided MAC Address FPMA)
In this
model this
is where
FC-2
functions
live
MAC Address of
FCoE_LEP
(VN_Port)
May or may not be
the same as the
FCoE controller
In this model this
is where the
Encapsulation
/De-Encapsulation
functions live
Multiple FC NPIV
instances on a single
logical FC Host interface
22
The Logical FC Link is defined by a MAC Address pair
A VN_Port MAC Address
A VF_Port MAC Address
For a logical FC link the FCoE Frames are always sent to and received from a specific FCFs
MAC Address
Therefore, pathing to and from the FC driver is always defined by the MAC Address of the
partner FCFs VF_Port
Multiple Logical FC connections
via a single Ethernet MAC
Examples of
single MACs
with
connections
to two
different
FCFs
Switch
23
Functions of an FCoE Initiator ASIC
NIC Function
F
C
o
E
f
u
n
c
t
i
o
n
FC
Function
Host PCIe interface
External
Port
FCoE
ASIC
Has a Normal NIC interface
(A) to the Host
Has one or more Normal
FC interfaces (B,C) to the
Host
FCoE functions not seen by
the Host
FCoE functions perform the
Encapsulation and De-
encapsulation
The FCoE function
Instantiates a Logical FC
N_Port, called a VN_Port
Lossless
Ethernet
MAC
A B C
24
HBA with Multiple Logical FC Interfaces
MAY have one burnt-in
MAC address for both IP
and FCoE/FIP packets
Or
MAY have different Burnt-
in MACs for IP and
FCoE/FIP packets
Used to separate HW
based FCoE from other
Ethernet Traffic
Most NICs come with
several Burnt-in MAC
Addresses
The FCoE controller will perform the FIP functions and will instantiate new
VN_Ports as FCoE Link End Point (LEP)
With the same MAC address as the FCoE Controller (SPMA)
Or with a new MAC address specified by the FCF (FPMA)
F
C
o
E
C
o
n
t
r
o
l
l
e
r
F
C
-
3

/
F
C
-
4
s
F
C
E
n
t
i
t
y
V
N
_
P
o
r
t
F
C
o
E
E
n
t
i
t
y
F
C
o
E
_
L
E
P
Lossless Ethernet MAC
Ethernet_Port
NIC
Etype=
FCoE
or
FIP?
FCoE Chip
MAC
Address of
Burnt-in
MAC
FCoE Function
F
C
-
3

/
F
C
-
4
s
F
C
E
n
t
i
t
y
V
N
_
P
o
r
t
F
C
o
E
E
n
t
i
t
y
F
C
o
E
_
L
E
P
MAC
Address of
Burnt-in
MAC
A B C
25
FIP (FCoE Initiation Protocol)
Discovery and Link Instantiation
26
ENode to FCF Logical Links
FC
fabric
H1
H2
Classical
Ethernet
LAN
FCF B
Lossless
Ethernet
Switch
FCF A
VN_Ports,
VN_Port_Names
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEPs
VF_Port
FCoE_LEPs
VF_Port
VF_Port,
VF_Port_Name
FCoE_LEPs
VF_Port
FCoE_LEPs
VF_Port
FCF-MAC(A)
FCF-MAC(B)
MAC(H1)
MAC(H2)
FCF-MAC(x): A MAC address of a Lossless Ethernet port of FCF x
MAC(y): A MAC address of a Lossless Ethernet port of ENode y
Note: The FCFs must
discover each other and
create a Fabric before the
ENode Discover FCFs
An ENode must Locate FCFs with a Discovery protocol and then establish a Logical FC
Link with an FCF (VN_Port VF_Port) before an Normal FC frame flow
27
Initial Login Flow ladder (2 Phases)
End-Node
FCF
Discovery
M
u
ltic
a
s
t
to
A
L
L
F
C
F
s
Unicast FCF MAC Address

et.al. in J umbo Frame
FLOGI
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
FC Command
(Using the FCF
selected MAC
Address as the SA)
FC Command responses
(using the commands SA as a DA)
Discovery
Phase
Login Phase
Normal FC
Processing
FCoE Initialization
Protocol
FCOE Protocol
See Appendix for more details in Discovery Consideration and Actions
28
Subsequent Login Flow ladder (1 Phase)
End-Node
FCF
FLOGI
Login Phase
Normal FC
Processing
FCoE Initialization
Protocol
FCOE Protocol
See Appendix for more details in Discovery Consideration and Actions
FC Command
(Using the FCF
selected MAC
Address as the SA)
FC Command responses
(using the commands SA as a DA)
FLOGI ACC with the FCFs
chosen VN_Port MAC address
as a descriptor value
29
Topologies
30
FCoE Switch & FC Fabric
FC
FC FC
FCoE Switch
An FCoE Switch may connect to a normal FC switch
Via the FC E-Port
Ethernet
FC
Note: FCoE servers
and storage will
probably use an FCoE
HBA (or chip set)
31
Multiple FCoE Switch (FCF) Topologies
Ethernet
FC
FC
FC FC
Lossless
Ethernet
Fabric
FC
FC FC
Lossless Ethernet
Fabric
FCoE Switch
A Lossless Ethernet Fabric can be made up of Combo FCoE Switches
Lossless Ethernet switches configured into a Lossless Ethernet
Fabric can Front the FCoE Switch
FC
FC FC
FCoE
Switch
FCoE
Switch
FCoE
Switch
Lossless Ethernet
FCoE Switches deployed at the edges of the Lossless Ethernet Fabric
FCoE Switches connected via VE_Ports and Lossless Ethernet
FC
FC FC
FCoE
Switch
FCoE
Switch
Lossless
Ethernet
1
Lossless
Ethernet
2
Lossless
Ethernet
4
FCoE
Switch
Lossless
Ethernet
3
A VE_Port in an FCF may connect to other VE_Port in another FCF
And an FCF FC E_Port may connect to an FC switch E_Port
32
Status of Current Data Center
Networking
There can be 3 different
networks
System Area Network (SyAN)
Used for Clustering/Low Latency
Storage area Network (SAN)
Used to Access to Storage
LAN/WAN External Networks
Used for General messaging
Used for Client-Server Messaging
Used for NAS
Often divided into at least 3
management domains
Data Center Server (clustering)
Network
Data Center Storage Network
Outfacing (IP) Network
LAN/WANs
Messaging
NAS
Data Center
Remote Offices
System Area Network (SyAN)
Clustering Fabric
InfiniBand
Myrinet
Ethernet
CTC
Data Center
Server & Storage Network
Management Group
Management Group
LAN/WAN
Messaging
NAS
Storage Area Network
- Fibre Channel
Note: with multiple Data Centers there may also
be interconnects with DWDM, FCIP/iFCP, etc.
33
Current Fabrics
Remote Remote
Offices Offices
Clustering Network
There is a FC Storage Network, a Clustering Network & an Outfacing Network
IP Network Management Group
FC
FC
FC
Ficon
Ficon Storage
Controller
Mainframe
Data Center Server & Storage Network Management Group
File Storage
Arrays (NAS)
iSCSI
Storage
FC
Network
Ficon
(LAN/WAN) (LAN/WAN)
Messaging Messaging
NAS NAS
Outfacing
IP Network
Local & Remote Local & Remote
Business Campus Business Campus
Focus: Low
Latency & High
Bandwidth
Focus: Protection,
Bandwidth/Congestion
Management
FC Link
Ethernet
Link
Ficon
Link
Clustering Network
Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.
34
View of the Future Data Center
Networks
Remote Remote
Offices Offices
Data Center Server &
Storage Network
Management Group
Management Group
Storage & System Storage & System
Area Network Area Network
Logically Single Fabric Logically Single Fabric
(with FCoE) (with FCoE)
Other IP Nets Other IP Nets
(LAN/WAN) (LAN/WAN)
Messaging Messaging
NAS NAS
The The Server/Storage Networks Server/Storage Networks
will become a Consolidated will become a Consolidated
Fabric Fabric
Managed by the Data Center Managed by the Data Center
System and Storage System and Storage
Management Group Management Group
Includes Storage and Includes Storage and
clustering provisioning clustering provisioning
SAN and SyAN managed SAN and SyAN managed
as a single fabric as a single fabric
Focus: High Bandwidth and Focus: High Bandwidth and
Low Latency Low Latency
The Outfacing ( The Outfacing (IP) Management IP) Management
Group Group remains the same remains the same
Focus: Protection, Bandwidth Focus: Protection, Bandwidth
and Congestion Management and Congestion Management
Note: with multiple Data Centers there may also
be interconnects with DWDM, FCIP/iFCP, etc.
35
Phase in: FC & High Performance Lossless
Ethernet Unify into a Data Center Fabric
Remote Remote
Offices Offices
FC
FC
FCoE
FC
(LAN/WAN) (LAN/WAN)
Messaging Messaging
NAS NAS
Outfacing
IP Network
Ficon
iSCSI
Storage
Ficon
Ficon Storage
Controller
Mainframe
File Storage
Arrays (NAS)
FCoE permits intermixing of multiple Connection types/protocols
Clustering messaging, General Messaging, and Storage
The DataCenter Fabric will Trunk to the Outfacing Network (including iSCSI sys)
Some Customers may want keep a mixed environment on-going
DataCenter
Fabric
Ethernet
SW
Ethernet
SW
FC & Ethernet
SW Blades
FC Link
Ethernet
Link
Ficon
Link
Business
Campus
with iSCSI
connections
Including iSCSI
Gateways
Note: with multiple Data Centers
there may also be interconnects
with DWDM, FCIP/iFCP, etc.
36
Scenarios
37
Scenario 1: FCoE & IP Flows
FC
FC FC
Lossless
Ethernet
Classical
Ethernet
FCoE
Switch
Internet
FCoE
Switches
Lossless
Ethernet
FCoE Flows
IP Flows
38
Scenario 2: FCoE Right & Wrong
FC
FC FC
Lossless
Ethernet
Classical
Ethernet
FCoE
Switch
Internet
FCoE
Switches
Lossless
Ethernet
FCoE Flows
Invalid FCoE Flows
39
Summary
40
Summary
FCoE is a simple, efficient mechanism for
encapsulating Fibre Channel in Ethernet frames
FCoE is being standardized in INCITS Fibre
Channel (T11) technical committee
Target completion is 2H08
Maximum benefit of Fibre Channel is achieved:
Evolutionary model of FC Switches and FC SANs
Emphasis placed on capitalizing on the benefits of
Converged Enhanced Ethernet (CEE)
Being discussed in the IEEE 802.1 standards working group
Thank You!
42
Q&A / Feedback
Please send any questions or comments on this
presentation to SNIA: tracknetworking@snia.org
For additional information refer to
http://www.t11.org/fcoe
Many thanks to the following individuals
for their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti Howard Goldstein Walter Dey
Robert Snively Suresh Vobbilisetty Silvano Gai
Joe Pelissier John Hufferd
43
Appendix
Additional
Requirements
FCoE Relation to ISO
Layers
Flows
Additional Topologies
FSPF and STP
FIP Considerations
and Actions
FIP Descriptors
Pause vs. BB_Credit
44
FCoE must be a direct mapping of Fibre Channel over
an Ethernet network
FCoE must be layered on top of Ethernet
FSPF used to route FCoE packets
Ethernet Spanning Tree (STP), MSTP, etc, is at a layer below
FCoE to allow an evolutionary approach towards
consolidation of fabrics
The Fibre Channel N_Port, F_Port and E-Port constructs must
be retained
With FCoE, ports may be connected with Logical Ethernet Links
May pass through Ethernet switches
Identified by pairs of end point MAC addresses
Physical Ethernet Links can replace physical FC Links
Physical Ethernet Links can carry all Ethernet traffic, including
FCoE, but combined traffic needs the CEE capabilities
45
Combo FCoE Switches may be built that support
normal Ethernet traffic, FCoE traffic , & FC traffic
The FCoE solutions should appear as a Fibre Channel to
a Fibre Channel experienced customer
FCoE should keep the Fibre Channel operations
independent from Ethernet forwarding
Keeps management /Troubleshooting simple
Common physical structures, different logical structures
Based on Ethertype (Ethertype = FCoE)
Storage Management should be unchanged
46
FCoE Relation to ISO Layers
47
Flows
48
Logical Fabric Topology
FC
FC FC
FCoE
Switch
FCoE
Switch
FCoE
Switch
Lossless
Ethernet
Lossless
Ethernet
A
H1
H2
H3
H4
S2
Ethernet
Destination
& Source
Encapsulated
FC Frame
D_ID
S_ID
FCoE-A MAC
FCoE-H2 MAC
FC_ID for S1
FC_ID for H2
FC_ID for S1
FC_ID for H2
FC_ID for S1
FC_ID for H2
FC_ID for S1
FC_ID for H2
FCoE-B MAC
FCoE-A MAC
FCoE-C MAC
FCoE-B MAC
FCoE-S1 MAC
FCoE-C MAC
Ethernet
FC
Logical
Transaction Path
An FCoE Switch receives FCoE frames addressed to its FC-MAC address and
forwards them based on the D_ID of the encapsulated FC frame
An FCoE Switch rewrites the SA and DA of an FCoE frame
Lossless
Ethernet
Lossless
Ethernet
S1
Path #1
Path #2 Path #3 Path #4
#1
#2
#3
#4
B
C
49
Single Ethernet Fabric with FCoE Switches
FC
FC FC
FCoE
Switch
FCoE
Switch
Lossless Ethernet
A
C
H1
H2
H3
S1
S2
FCoE-A MAC
Ethernet
Destination
& Source
FCoE-H2 MAC
Encapsulated
FC Frame
D_ID
S_ID
FC_ID for S1
FC_ID for H2
FC_ID for S1
FC_ID for H2
FC_ID for S1
FC_ID for H2
FCoE-C MAC
FCoE-A MAC
FCoE-S1 MAC
FCoE-C MAC
Ethernet
Switch
Ethernet
Switch
Ethernet
Switch
Ethernet
Switch
Ethernet
Switch
Ethernet
FC
Logical
Transaction Path
Path #1 Path #2 Path #3
#1
#2
#3
50
FC Host to FCoE Storage
FC
FC FC
FCoE
Switch
FCoE
Switch
Lossless Ethernet
H1
H2
H3
S1
S3
FCoE-S2 MAC
Ethernet
Destination
& Source
FCoE-A MAC
Encapsulated
FC Frame
D_ID
S_ID
FC_ID for S2
FC_ID for H5
FC_ID for S2
FC_ID for H5
FC_ID for S2
FC_ID for H5
FCoE-A MAC
FCoE-C MAC
Ethernet
Switch
Ethernet
Switch
Ethernet
Switch
Ethernet
Switch
H5
S2
Ethernet
Switch
Ethernet
FC
Logical
Transaction Path
Path #3 Path #2
Path #1
#1
#2
#3
C
A
51
Additional Topologies
52
Additional Topologies (1)
FCoE
Switch (A)
FCoE
Switch (B)
FCoE
Switch (E)
FCoE
Switch (F)
Lossless
Ethernet
Switch
Lossless
Ethernet
Switch
FC
FC FC
Example of Topologies with Rack Mount servers
53
Additional Topologies (2)
FCoE
Switch (B)
FCoE
Switch (E)
FCoE
Switch (F)
Lossless
Ethernet
Switch
Lossless
Ethernet
Switch
FC
FC FC
FCoE
Switch (A)
Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) Support
Example of Topologies with Blade Servers
54
FSPF and STP
55
FSPF & STP Concepts with FCoE (basic)
56
FSPF & STP Concepts with FCoE (interconnected)
57
Equivalent FC topology
58
FIP Considerations and Actions
59
FCoE Discovery Considerations
The Discovery phase of the FCoE Initialization Protocol (FIP)
uses two types of messages, Solicitations and Advertisements
The FIP Discovery Phase helps define the FCF Ports that are
available for the Link instantiation Phase
The ENodes discover the FCF ports that can become VF_Ports
and FCFs discover other FCF ports that can become VE_Ports
ENodes Solicit (via Multicast) Advertisements from FCFs while
specifying their capabilities
In response FCFs Advertise their availability and capabilities back
to the ENodes
FCFs Multicast their existent to other FCF
The FIP Discovery phase exchanges solicitation and/or
Advertisements between (HBA and/or FCF) FCoE Controllers
60
FCF Discovery Actions
FCF supporting VE_Ports:
Discovers other VE_Port capable FCF-MACs, connected
to the same Lossless Ethernet segment, by:
Transmitting a multicast Solicitation to All-FCFs
(with the FCF bit set to one)
Receives back J umbo Unicast Advertisements from VE capable
MACs
To verify the support of Ethernet Jumbo frames in the path
In response to receipt of a Multicast to All-FCFs from another
FCF
Has its own VE capable MACs sends their own J umbo Unicast
Advertisements
Instantiates VE_Port to VE_Port connections and
Exchanges FC ELP (Extended Link Protocol) and Fabric
configuration (using Ethertype=FCoE) with the other
FCFs VE_Port capable MACs
61
ENode Actions
When an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACs
connected to the same Lossless Ethernet segment
Transmits a multicast Solicitation to All-FCFs (with the FCF bit set
to zero)
Receives J umbo unicast Advertisements from select compatible
VF_Port capable FCF-MACs
Stores the discovered FCF-MACs in an FCF list
When an ENode receives an Advertisement that a
new FCF is available, it may send a unicast
Solicitation to it and receive a Jumbo unicast
Advertisement in reply
To verify the support of Ethernet Jumbo frames in the path
May then perform FLOGIs (with Ethertype-FIP) to a
vendor specific subset of the FCF-MACs in the FCF
list
62
FIP Descriptors
63
FIP Descriptors (1)
64
FIP Descriptors (2)
65
Pause vs. BB_Credit
Both mechanisms are used to avoid dropping frames
With different trade-offs
The Pause mechanism requires at least the (2 x RTT x
bandwidth) product on a link as buffer space
But allows Buffer handling in an arbitrary way
Well suited for networks with limited (bandwidth x delay) product
(e.g. within the data center)
The Pause frame is handled by the MAC layer
Similar to the R_RDY handling by the FC-1 level
The BB_Credit mechanism prevents loosing frames
over any link
But links go under-utilized if link credits (& buffers) are < that
needed for (RTT x BW)
Requires buffer handling in maximum frame size units

Fibre Channel Over Ethernet

Uploaded by

Copyright:

Available Formats

You might also like

Fibre Channel Over Ethernet

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Fibre Channel Over Ethernet

Uploaded by

Copyright:

Available Formats

Fibre Channel over Ethernet (FCoE)

John L Hufferd, Brocade

Unicast FCF MAC Address

You might also like