Brksan 2821

BRKSAN-2821 14572_05_2008_c1
2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
I/O Consolidation in the Data Center
BRKSAN-2821
BRKSAN-2821 14572_05_2008_c1
Cisco Public
2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr
Agenda
Section 1: What is I/O Consolidation Section 2: Enabling Technologies Section 3: FCoE (Fibre Channel over Ethernet) Section 4: I/O Consolidation Use Cases Challenges Closing Remarks
BRKSAN-2821 14572_05_2008_c1
Cisco Public
Section 1 What Is I/O Consolidation
BRKSAN-2821 14572_05_2008_c1
Cisco Public
What Is I/O Consolidation

IT organizations operate multiple parallel networks
IP and other LAN protocols over an Ethernet network SAN over a Fibre Channel network HPC/IPC over an InfiniBand network
I/O consolidation supports all three types of traffic onto a single network Servers have a common interface adapter that supports all three types of traffic
IPC: Inter-Process Communication

BRKSAN-2821 14572_05_2008_c1 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
I/O Consolidation in the Network
Processor Memory
Processor Memory
I/O
Storage
I/O
I/O
I/O Subsystem
Storage
LAN
IPC
IPC: Inter-Process Communication

LAN
IPC
I/O Consolidation in the Host

Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs Limited number of interfaces for Blade Servers
FC HBA FC HBA NIC NIC NIC HCA HCA
BRKSAN-2821 14572_05_2008_c1
FC Traffic FC Traffic
CNA
Enet Traffic Enet Traffic Enet Traffic IPC Traffic IPC Traffic
Cisco Public
CNA
All Traffic Goes over 10 GE
Cabling and I/O Consolidation
BRKSAN-2821 14572_05_2008_c1
Cisco Public
I/O Consolidation: Benefits to Customers

FC Traffic FC Traffic
FCoE FCoE
FCoE
SAN A SAN B
Enet Traffic Enet Traffic
FCoE
FCoE SAN
Fewer CNAs and Cables
Storage Keeps the Same Management Model as Native FC
Display FCoE Adapter
FC Storage
FC Switch
FCoE Switch
Server
No Storage Gateway
BRKSAN-2821 14572_05_2008_c1
Less Power and Cooling
Cisco Public
Merging the Requirements

LAN/IP
Must be Ethernet
Too much investment Too many applications that assume Ethernet
Storage
Must follow the Fibre Channel model Losing frames is not an option
IPC
(Inter-Process Communication)
Doesnt care of the underlying network, provided that:

It is cheap It is low latency It supports APIs like OFED, RDS, MPI, sockets
BRKSAN-2821 14572_05_2008_c1
Cisco Public
10
Why Has It Not Succeeded Yet?

Previous attempts
Fibre Channel Never credible InfiniBand Not Ethernet iSCSI Not Fibre Channel
Before PCI-Express there was not enough I/O bandwidth in the servers It needs to be Ethernet, but
1 GE didnt have enough bandwidth
11
PCI-Express
PCI Express (PCI-E or PCIe)
A computer expansion card interface format designed to replace PCI, PCI-X, and AGP
PCIe 1.1
Serial links at 2.5 Gbps (2 Gbps at the Datalink) Speeds from 2 Gbps (1x) to 32 Gbps (16x) 8x is required for 10 GE
PCIe 2.0 (aka PCIe Gen 2)

Doubles the bandwidth per serial link from 2 Gbit/s to 4 Gbit/s Spec available since January 2007 Products are making their way into the market
12
10 GE
2008 will be the year of 10GE 10 GE has enough bandwidth Merging example
2 x 1 GE Ethernet NIC 1 x 4 Gbps FC (really 3.2 Gbps) Total 5.2 Gbps over a 10 Gbps link
CNAs will all be dual-ported for HA

20 Gbps usable bandwidth per server with a single CNA
BRKSAN-2821 14572_05_2008_c1
Cisco Public
13
Evolution of Ethernet Physical Media

Role of Transport in Enabling These Technologies
Mid-1980s 10 Mb Mid-1990s 100 Mb
UTP Cat 5
SFP+ to SFP+
Early-2000s
Late-2000s 10 Gb
SFP+ 1 Gb Cu
UTP Cat 5 SFP Fiber
Technology SFP+ CU Copper SFP+ USR

Ultra Short Reach
Cable Twinax MM OM2 MM OM3 MM 62.5 m MM 50 m

Cat6 Cat6a/7 Cat6a/7
Distance 10 m 10 m 100 m 82 m 300 m

55 m 100 m 30 m
Power (Each Side) 0.1W
Transceiver Latency (Link) 0.1 s
1W
0 0
2.5 s 2.5 s 1.5 s
SFP+ SR
Short Reach
1W
8W 8W 4W
10GBASE-T
BRKSAN-2821 14572_05_2008_c1
Cisco Public
14
Twin-ax Copper Cable

Low power consumption Low cable cost Low transceiver latency
Application Server
SAN A
LAN
SAN B
Low error rate (1017)

16x10 GE Cables
Application Server Application Server
Application Server Application Server Application Server Application Server Application Server Application Server Application Server Application Server
Easier to manage cabling solution reduces deployment time All copper cables are contained within rack
Application Server Application Server Application Server Application Server
16x10 GE Cables
Thinner cable with higher bend radius
Application Server
15
Drivers for 10GE to the Servers

Multicore CPU Architectures Allowing Bigger and Multiple Workloads on the Same Machine
Server Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation
Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server
Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections
BRKSAN-2821 14572_05_2008_c1
Cisco Public
16
Section 1 Enabling Technologies
BRKSAN-2821 14572_05_2008_c1
Cisco Public
17
Three Challenges + One
BRKSAN-2821 14572_05_2008_c1
Cisco Public
18
Why Are Frames Lost?

Collision
No longer present in full duplex Ethernet
Transmission Error
Very rare in the data center
Congestion
Most common cause Congestion is a switch issue, not a link issue
A full duplex IEEE 802.3 link does not lose frames
It must be dealt with in the bridge/switch

By IEEE 802.1
BRKSAN-2821 14572_05_2008_c1
Cisco Public
19
Can Ethernet Be Lossless?

Yes, with Ethernet PAUSE Frame
Ethernet Link
STOP
Switch A
PAUSE
Queue Full
Switch B
Defined in IEEE 802.3Annex 31B

The PAUSE operation is used to inhibit transmission of data frames for a specified period of time
Ethernet PAUSE transforms Ethernet into a lossless fabric

20
10
How PAUSE Works

Threshold
A PAUSE Frame Start Sending Stop Frames for This Frames Again Interval of Time
21
Lets Compare PAUSE with FC Buffer to Buffer Credit

Eight credits preagreed
A
R_RDY
BRKSAN-2821 14572_05_2008_c1
Cisco Public
22
11
How PAUSE Propagates

Threshold
PAUSE
PAUSE
S1
S2
S3
BRKSAN-2821 14572_05_2008_c1
Cisco Public
23
PAUSE Frame Format

PAUSE Frame
01:80:C2:00:00:01
A standard Ethernet frame, not tagged EtherType = 0x8808 means MAC Control Frame Opcode = 0x0101 means PAUSE Pause_Time is the time the link needs to remain paused in Pause Quanta (512-bits time) There is a single Pause_Time for the whole link
Source Station MAC EtherType = 0x8808 Opcode = 0x0001 Pause_Time
Pad 42 Bytes
CRC
BRKSAN-2821 14572_05_2008_c1
Cisco Public
24
12
Why Is PAUSE Not Widely Deployed?

Inconsistent implementations
Standard allows for asymmetric implementations Easy to fix
PAUSE applies to the whole links

Single mechanism for all traffic classes
This may cause traffic interference

e.g., Storage traffic paused due to a congestion on IP traffic
BRKSAN-2821 14572_05_2008_c1
Cisco Public
25
Priority Flow Control (PFC)

a.k.a. PPP (Per Priority Pause) PFC enables PAUSE functionality per Ethernet priority
IEEE 802.1Q defines eight priorities Traffic classes are mapped to different priorities: No traffic interference IP traffic may be paused while storage traffic is being forwarded Or, vice versa Requires independent resources per priority (buffers)
High level of industry support

Cisco distributed proposal Standard track in IEEE 802.1Qb
16
EtherType = IEEE 802.1Q
12 Bits
VLAN ID
Priority CFI
IEEE 802.1Q Tag

26
13
Priority Flow Control in Action

Transmit Queues Ethernet Link
One Two Three Four Five Six Seven Eight
STOP PAUSE
Receive Queues
One Two Three Four Five Six Seven Eight
Eight Priorities
Switch A
Switch B
27
PFC Frame Format

Priority Flow Control
01:80:C2:00:00:01
Similar to the PAUSE frame Opcode = 0x0101 is used to distinguish PFC from PAUSE Class vector indicates for which priorities the frame carries valid Pause information There are eight Time fields, one per priority
Source Station MAC EtherType = 0x8808

Opcode = 0x0101 Class Enable Vector Time (Class 0) Time (Class 1) Time (Class 2) Time (Class 3) Time (Class 4) Time (Class 5) Time (Class 6) Time (class 7)
Pad 28 Bytes
CRC
BRKSAN-2821 14572_05_2008_c1
Cisco Public
28
14
Is Lossless Better?
Frames are not dropped FC over lossless Ethernet works well
TCP relies on losses We can run it on a priority where we do not enable Pause
Congestion spreading and head of line blocking
BRKSAN-2821 14572_05_2008_c1
Cisco Public
29
Is Anything Else Required?

In Order to Build a Deployable I/O Consolidation Solution, the Following Additional Components Are Required:
Discovery protocol (DCBX) Bandwidth manager Congestion management
BRKSAN-2821 14572_05_2008_c1
Cisco Public
30
15
Discovery Protocol
DCBX: Data Center Bridging eXchange
Data Center Ethernet Links DCBX Data Center Ethernet Links with Partial Enhancements
DCBX
Legacy Ethernet Links Legacy Ethernet Network

XP
Data Center Ethernet Cloud

BC DC
BRKSAN-2821 14572_05_2008_c1
Cisco Public
31
DCBX
Hop-by-hop negotiation for:
Priority Flow Control (PFC) Bandwidth management Congestion management (BCN/QCN) Applications Logical link-down
Based on LLDP (Link Level Discovery Protocol)

Added reliable transport
Allows either full configuration or configuration checking

Link partners can choose supported features and willingness to accept configuration from peer
32
16
Bandwidth Management
IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism Products typically implement some form of Deficit Weighted Round Robin (DWRR)
Configuration and interworking is problematic
Proposal for HW-efficient, two-level DWRR with strict priority support

Consistent behavior and configuration across network elements
Standard track in IEEE 802.1Qaz
BRKSAN-2821 14572_05_2008_c1
Cisco Public
33
Priority Groups
Priority Groups Are Then Scheduled LAN
Priorities Are Assigned to Individual Traffic Classes
SAN
IPC Priority Groups First Level of Scheduling Inside Each Group

Final Link Behavior
34
17
Goals
BW assignment for each Priority Group
Example: 40% LAN, 40% SAN, 20% IPC
Cannot compromise lowlatency application due to convergence

Allow strict, high priority scheduling of IPC (and equivalent) traffic
Should allow multiple traffic classes within a Priority Group

Allow these traffic classes to share BW without hard configuration Example: VoIP and bulk traffic to share 40% LAN BW
Should provide management infrastructure (MIBs)

Defining scheduling algorithms is too restrictive and not necessary Interoperability for management is important
BRKSAN-2821 14572_05_2008_c1
Cisco Public
35
Example of Link Bandwidth Allocation

Offered Traffic
3 Gbs 3 Gbs 2 Gbs
10 GE Link Realized Traffic Utilization (30%) HPC Traffic (30%) LAN Traffic (40%) (20%)
3 Gbs
4 Gbs
6 Gbs
(30%)
(50%)
3 Gbs
3 Gbs
3 Gbs
(30%)
Storage Traffic (30%) T2
(30%)
T1
T2
T3
T1
T3
HPC TrafficPriority Class High20% Guaranteed Bandwidth LAN TrafficPriority Class Medium50% Guaranteed Bandwidth Storage TrafficPriority Class Medium-High30% Default Bandwidth
36
18
Congestion Management
Layer 2, end-to-end congestion management Standards track in IEEE 802.1Qau a.k.a. BCN (Backward Congestion Notification) or QCN (Quantized Congestion Notification)
Switch Switch
Switch Transmit Queue Switch Switch
Thro ttle
Receive Buffer
e ottl Thr
BRKSAN-2821 14572_05_2008_c1
Cisco Public
37
Congestion Management Principles

Move congestion to network edges to avoid congestion spreading Use rate-limiters at the edge to shape flows causing congestion
Tune rate-limiter parameters based on feedback coming from congestion points
Inspired by:
TCP AIMD (Additive Increase, Multiplicative Decrease) rate control TCP window increases linearly in absence of congestion Decreases exponentially (gets halved) at every congestion indication (either implicit or explicit) FCC (Fibre Channel Congestion Control) A feature on Cisco MDS switches
38
19
Congestion Point and Reaction Point

Roles and responsibilities
Reaction Points (RP) shape traffic entering the network Congestion Points (CP) indicate congested state of queuing points
RP
RP
CP
BRKSAN-2821 14572_05_2008_c1
Cisco Public
39
DCB: Data Center Bridging

Industry consensus term to indicate
Priority flow control Bandwidth management Congestion management Discovery (DCBX)
BRKSAN-2821 14572_05_2008_c1
Cisco Public
40
20
DCE: Data Center Ethernet

Cisco term used to indicate Cisco switches that implement the DCB features, plus
Layer 2 multipathing Fibre Channel over Ethernet
BRKSAN-2821 14572_05_2008_c1
Cisco Public
41
Layer 2 Multipathing
Increase bandwidth of L2 networks via multiple active links
BRKSAN-2821 14572_05_2008_c1
Cisco Public
42
21
L2 Multipathing
Multiple paths are used, reclaiming network bandwidth L3 multipathing is common in IP networks Important when there is limited or no differentiation in speed between access links and backbone links Reduces latency L2 multipathing
Eliminates Spanning Tree from the backbone No packet flooding Small forwarding tables
BRKSAN-2821 14572_05_2008_c1
Cisco Public
43
Layer 2 Multipathing
Cisco DCE is:
A precursor of TRILL, an IETF project for Layer 2 multipath Inspired to FSPF (Fibre Channel Shortest Path First)
Cisco DCE
Computes topology and forwarding via IS-IS Provides optimal pair-wise unicast forwarding Provides multipathing for unicast and multicast frames Provides seamless interoperability with existing devices
BRKSAN-2821 14572_05_2008_c1
Cisco Public
44
22
FCoE: Fibre Channel over Ethernet

FCoE is the protocol used to carry Fibre Channel over CEE/DCE
Allows storage I/O consolidation
FCoE
SAN
LAN
SAN
Its in an advanced state of definition in INCITS T11 FC-BB-5 WG
BRKSAN-2821 14572_05_2008_c1
Cisco Public
45
Delayed Drop
Delayed Drop is a mechanism that:
Allows a switch buffer to virtually extend to previous hop This reduces packet drop for transient congestions Is enabled per priority
It is implemented by asserting PFC on the priority for a short time

After that time, traffic can flow again or can be dropped Delayed Drop Is a Means of Using PFC to Mitigate the Effects of Short-Term Traffic Bursts While Maintaining Packet Drop for Long-Term Congestion
BRKSAN-2821 14572_05_2008_c1
Cisco Public
46
23
Delay Drop and Proxy Queue

Actual Queue Traffic Flow
PAUSE
Proxy Queue
UNPAUSE
During short-term congestion, both queues drain fast enough that the actual queue releases the PAUSE on its own During long-term congestion, the proxy queue fills to its high-water mark, and it releases the PAUSE; the actual queue begins dropping packets, and the congestion is managed through higherlevel protocols
47
Section 3 FCoE: Fibre Channel over Ethernet
BRKSAN-2821 14572_05_2008_c1
Cisco Public
48
24
FCoE: FC over Ethernet

FCoE is I/O consolidation of FC storage traffic over Ethernet
FC traffic shares Ethernet links with other traffics Requires a lossless Ethernet fabric
Ethernet Fibre Channel Traffic
BRKSAN-2821 14572_05_2008_c1
Cisco Public
49
FCoE Protocol Stack

From a Fibre Channel standpoint, its FC connectivity over a new type of cable called an Ethernet cloud From an Ethernet standpoint, its yet another ULP (Upper Layer Protocol) to be transported
1, 10Gbps
BRKSAN-2821 14572_05_2008_c1
Cisco Public
50
25
FCoE Benefits
FCoE benefits are the same of any I/O consolidation solution
Fewer cables Both block I/O and Ethernet traffic coexist on same cable Fewer adapters needed Overall less power
Plus additional advantages of being FC

Seamless integration with existing FC SANs No gateway
BRKSAN-2821 14572_05_2008_c1
Cisco Public
51
FCoE Benefits
LAN
4
SAN-A
SAN-B
LAN
4
SAN-A
SAN-B
Nearly Half the Cables

16 Servers Adapters Switches Cables Mgmt Pts
BRKSAN-2821 14572_05_2008_c1
Enet 16 2 36 2
FC 16 2 36 2
Total 32 4 72 4
Cisco Public
16 Servers Adapters Switches Cables Mgmt Pts
Enet 16 2 36 2
FC 0 0 4 0
Total 16 2 40 2
52
26
FCoE Is Fibre Channel

FCoE Is Fibre Channel at the Host and Switch Level
Easy to Understand Same Operational Model Same Techniques of Traffic Management Same Management and Security Models Completely Based on the FC Model Same Host-to-Switch and Switch-to-Switch Behavior of FC e.g., in Order Delivery or FSPF Load Balancing WWNs, FC-IDs, Hard/Soft Zoning, DNS, RSCN
BRKSAN-2821 14572_05_2008_c1
Cisco Public
53
Protocol Organization
FCoE Itself Is the data plane protocol It is used to carry most of the FC frames and all the SCSI traffic
FIP (FCoE Initialization Protocol) It is the control plane protocol It is used to discover the FC entities connected to an Ethernet cloud It is also used to login to and logout from the FC fabric
The Two Protocols Have:

Two different EtherTypes Two different frame formats
54
27
FCoE Frame Size

Ethernet Header FCoE Header
12 Bytes (MAC Addresses) + 4 Bytes (802.1Q Tag) 16 Bytes
FC Header
24 Bytes
Tota l:
Up to 2112 Bytes 4 Bytes
2180 Byte s
CRC EOF FCS

BRKSAN-2821 14572_05_2008_c1 2008 Cisco Systems, Inc. All rights reserved.
FCoE Frame Format

Destination MAC Address
EOF FCS
FC Payload
ET = FCoE
1 Byte (EOF) + 3 Bytes (Padding) 4 Bytes

Cisco Public
55
Source MAC Address IEEE 802.1Q Tag Ver Reserved Reserved Reserved Encapsulated FC Frame (Including FC-CRC) Reserved SOF Reserved
56
28
ENode: Simplified Model

ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC
a.k.a. CNA (Converged Network Adapter)
FC Node FCoE FCoE
Enet Port
Enet Port
BRKSAN-2821 14572_05_2008_c1
Cisco Public
57
FCoE Switch: Simplified Model

FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch
FC Port
FCoE Switch FCF FCoE
FC Port FC Port FC Port Eth Port Eth Port Eth Port
Ethernet Bridge
Eth Port
Eth Port
Eth Port
Eth Port
Eth Port
BRKSAN-2821 14572_05_2008_c1
Cisco Public
58
29
FC-BB-5 Terminology
Unchanged from previous FC standard
VN_Port: Virtual N_Port VF_Port: Virtual F_Port VE_Port: Virtual E_Port
Added to support FCoE

FCoE_LEP (FCoE link endpoint): The data forwarding component that handles FC frame encapsulation/decapsulation, and transmission/reception of FCoE frames FCoE Controller: the entity that implement the FIP protocol
BRKSAN-2821 14572_05_2008_c1
Cisco Public
59
ENode: Complete Model
FC-3/FC-4s
FC-3/FC-4s
FC-3/FC-4s
FC-3/FC-4s
VN_Port
FC Entity
VN_Port
FC Entity
VN_Port
FC Entity
VN_Port
FC Entity
FCoE_LEP
FCoE Entity FCoE Controller
FCoE_LEP
FCoE Entity
FCoE_LEP
FCoE_LEP
FCoE Entity
Lossless Ethernet MAC
Ethernet_Port
Ethernet_Port
SC DI FLOGI F
60
30
FCoE Switch: Complete Model

FC Fabric Interface
E_Port E_Port E_Port F_Port F_Port
Means Optional
F_Port
FC Switching Element
VE_Port
FC Entity
VF_Port
FC Entity
VE_Port
FC Entity
VF_Port
FC Entity
FCoE_LEP
FCoE_LEP FCoE FCoE_LEP Entity FCoE_LEP
FCoE_LEP
FCoE_LEP FCoE FCoE_LEP Entity FCoE_LEP
Ethernet_Port
Ethernet_Port
Lossless Ethernet Bridging Element

Ethernet_Port Ethernet_Port Ethernet_Port
Ethernet_Port
Lossless Ethernet Bridging Element

Ethernet_Port Ethernet_Port Ethernet_Port
Ethernet_Port
Ethernet_Port
Ethernet_Port
BRKSAN-2821 14572_05_2008_c1
Cisco Public
61
FCoE: Initial Deployment

SAN A SAN B 10 GE Backbone
VF_Ports VN_Ports
10 GE 4/8 Gbps FC
62
31
FCoE: Adding Blade Servers

VF_Ports
VN_Ports
10 GE 4/8 Gbps FC
63
FCoE: Adding Native FCoE Storage

VN_Ports VF_Ports
VN_Ports
10 GE 4/8 Gbps FC
64
32
FCoE: Adding VE_Ports

VE_Ports VF_Ports
VN_Ports
10 GE 4/8 Gbps FC
65
FCoE Addressing and Forwarding

FCoE frames have:
MAC addresses (hop-by-hop) FC addresses (end-to-end)
FC Storage
FCID 7.1.1 FC Fabric FC Domain 7 FC Fabric FC Domain 3 MAC A Ethernet Fabric FC Domain 1 MAC B Ethernet Fabric
FCID 1.1.1 MAC C
D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)
D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)
Dest. = MAC B Srce. = MAC A D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)
Dest. = MAC C Srce. = MAC B D_ID = FC-ID (1.1.1) S_ID = FC-ID (7.1.1)
FC Frame
FC Frame
FCoE Frame
BRKSAN-2821 14572_05_2008_c1
Cisco Public
66
33
FCoE MAC Addresses

VE_Ports and VF_Ports always use MAC addresses derived from the switch pool VN_Ports may use two types of MAC addresses
SPMA (Server Provided MAC Addresses) FPMA (Fabric Provided MAC Addresses)
MAC Addresses are negotiated in FIP Initial deployment will use FPMA only
BRKSAN-2821 14572_05_2008_c1
Cisco Public
67
The Mapped MAC Addresses

A dedicated MAC address for each FC-ID
Assigned by the FC fabrics Consistent with the FC model OUIs with U/L = 1 (Local addressing), called FC-MAPs Multiple FC-MAPs may be supported (one per FC fabric)
24 Bits FC-MAP (ex 02-12-34) 24 Bits FC-ID 7.8.9
MAC Address
BRKSAN-2821 14572_05_2008_c1
FC-MAP (ex 02-12-34) 48 Bits
FC-ID 7.8.9
Cisco Public
68
34
Initial Login Flow Ladder

ENode
Discovery
Solicitati on
Advertisement
FCoE Switch
Discovery FIP: FCoE Initialization Protocol
FLOGI/FDISC
FLOGI/FDISC Accept
FC Command
FC Command Responses
FCOE Protocol
BRKSAN-2821 14572_05_2008_c1
Cisco Public
69
FIP Frame: Contains FIP Operation

Destination MAC Address Source MAC Address IEEE 802.1Q Tag ET = FIP Ver Reserved
Encapsulated FIP Operation (Self-Describing Length) PAD to Minimum Length or Mini-Jumbo Length Ethernet FCS
BRKSAN-2821 14572_05_2008_c1
Cisco Public
70
35
FIP Descriptors (1)

Type = 1 Type = 2 Len = 4 Len = 8 MAC Address Type = 3 Len = 8 FC-MAP Type = 4 Len = 12 Switch_Name Type = 5 Len = 12 Fabric_Name
Reserved
Priority
Reserved
Reserved
Reserved
71
FIP Descriptors (2)

Type = 6 Len = 12 Port_Name Reserved
Type = 7
Len = XX
Reserved
FLOGI Request, FLOGI LS_ACC/LS_RJT NPIV FDISC Request, FDISC LS_ACC/LS_RJT Fabric LOGO Request, LOGO LS_ACC/LS_RJT (No SOF/EOF / FC-CRC?)
BRKSAN-2821 14572_05_2008_c1
Cisco Public
72
36
FIP
Multicast Solicitation from H2
MAC (H1) FCF-MAC (A)
H1 Lossless Ethernet Bridge
FCF A FC Fabric FCF A

MAC (H2) FCF-MAC (B)
H2
All-MACs MAC(H2) Solicitation (FIP)

[F=0, S=0, MAC(H2), Capability, Other]
Solicitation identifies VF_Port capable FCF-MACs with compatible addressing capabilities Other parameters may include ENodes Port_Name for optional duplicate MAC address detection
Cisco Public
BRKSAN-2821 14572_05_2008_c1
73
FIP
Unicast Advertisements from A and B

H2
MAC(H2) FCF-MAC(A) Mini-jumbo Advertisement (FIP)

[S=1, F=1, Priority, FC-MAP, FCF-MAC(A), Switch_Name, Fabric_Name, Capability, Other]
BRKSAN-2821 14572_05_2008_c1
MAC(H2) FCF-MAC(B)
H2s FCF list:

Mini-jumbo Advertisement (FIP)
[S=1, F=1, Priority, FC-MAP, FCF-MAC(B), Switch_Name, Fabric_Name, Capability, Other]
Cisco Public
FCF-MAC(A) [J] FCF-MAC(B) [J]
FCF not meeting capability of ENode does not reply
74
37
FIP
FLOGI Request

H2
FCF-MAC(A) MAC (H2) FLOGI Request (FIP)

[FC Header, FLOGI data, Proposed MAC(H2)]
FCF-MAC(B) MAC(H2) FLOGI Request (FIP)

[FC Header, FLOGI data, Proposed MAC(H2)]
Capability agreed during discovery
BRKSAN-2821 14572_05_2008_c1
Cisco Public
75
FIP
FLOGI LS_ACC

H2
MAC (H2) FCF-MAC(A) FLOGI LS-ACC (FIP)

[FC Header, LS_ACC data, Approved MAC(H2)]
MAC(H2) FCF-MAC(B) FLOGI LS-ACC (FIP)

[FC Header, LS_ACC data, Approved MAC(H2)]
ENode uses MAC address in FIP FLOGI LS_ACC as the VN_Port MAC address for the FC-ID contained in the FLOGI data for subsequent FCoE frames
76
BRKSAN-2821 14572_05_2008_c1
Cisco Public
38
FCoE
Data Transfer

MAC (H2) MAC (H2) MAC (H2) FCF-MAC (B)
H2
FCF-MAC(A) MAC(H2) Fibre Channel Frame (FCoE)

[FC SOF, FC Header, FC Data, FC CRC FC EOF]
BRKSAN-2821 14572_05_2008_c1
FCF-MAC(B) MAC(H2) Fibre Channel Frame (FCoE)

[FC SOF, FC Header, FC Data, FC CRC, FC EOF]
All subsequent FCoE frames use granted MAC address and assigned FC-ID
FIP frames continue to use MAC(H2) For SPMA, MAC(H2) = MAC(H2) = MAC(H2) For FPMA, MAC(H2) and MAC(H2) use FC-IDs as low order 24 bits and FC-MAP for upper 24 bits
77
Cisco Public
The Most Asked Question:

Is FCoE Routable?
FCoE
1, 2, 4, (8), 10 Gbps
1, 10 . . . Gbps
10, 20 Gbps
BRKSAN-2821 14572_05_2008_c1
Cisco Public
78
39
Is FCoE Routable?
Most folks mean Is FCoE IP-routable
The answer is NO, there is no IP layer in FCoE This was a design goal to keep FCoE simple FC-BB-5 contains FCIP that is IP-routable
FCoE is FC-routable
FCoE switches may forward FC frames across different Ethernet clouds FCoE switches may forward FC frames over the Internet using FCIP
FCoE
FCIP
IP Cloud
FCIP
FCoE
BRKSAN-2821 14572_05_2008_c1
Cisco Public
79
CNA: Converged Network Adapter

LAN
10GbE
Link
HBA
HBA HBA
CNA
10GbEE 10GbEE
BRKSAN-2821 14572_05_2008_c1
10GbE
Link
Link
Fibre Channel Ethernet
Fibre Channel Ethernet
PCIe
Ethernet Drivers
Ethernet
PCIe
PCIe
Fibre Channel Drivers
Ethernet Drivers
Fibre Channel Drivers
Operating System
2008 Cisco Systems, Inc. All rights reserved. Cisco Public
Operating System
80
40
View from Operating System

Standard drivers Same management Operating system sees:
Dual-port, 10 Gigabit Ethernet adapter Dual-port, 4 Gbps Fibre Channel HBAs
BRKSAN-2821 14572_05_2008_c1
Cisco Public
81
Open-FCoE Software
HBA HBA Mgmt Plane Linux Kernel File System layers SCSI Layer Linux Kernel File System layers SCSI Layer OpenFC Layer HBA Driver FCoE Layer Net Device HBA Ethernet Driver Ethernet FCoE FCoE Mgmt Plane
Fibre
BRKSAN-2821 14572_05_2008_c1 2008 Cisco Systems, Inc. All rights reserved.
Server
Cisco Public
Ethernet
Server
82
41
Open-FCoE Software: How to Get It

Open source project Open-FCoE.orgsource Git trees Or Open-FCoE source packageTBD Install a Linux Red Hat EL5, Fedora Core 7, or SuSE 10 distribution Update kernel to 2.6.23 or later Install: see Quick Start Guide at open-fcoe.org Use switch, soft-target, or gateway
BRKSAN-2821 14572_05_2008_c1
Cisco Public
83
Wireshark
Once known as Ethereal Captures and displays network traffic Available from: http://wireshark.org/ Sample trace file
/common/openfc/traces/fcoe-t11.cap
Use tcpdump to capture

tcpdump i eth0 s 0 w /tmp/fcoe.cap
Screenshots/demo
BRKSAN-2821 14572_05_2008_c1
Cisco Public
84
42
Wireshark Screenshot
BRKSAN-2821 14572_05_2008_c1
Cisco Public
85
Section 4 Case Studies
BRKSAN-2821 14572_05_2008_c1
Cisco Public
86
43
Current Data Center Environment

LAN Core SAN-A SAN-B
Distribution
4 MDS 9500 Cisco Catalyst 6509
4 x 4G FC
1 x 10 GE
STP BLK
POD 1
POD N
Access
MDS 9500 Cisco Catalyst 6509 NIC Teaming Active/Standby Discrete 1 GE NICs and FC HBA
Server Cabinet 1
Server Cabinet N
Server Cabinet 1
Server Cabinet N
BRKSAN-2821 14572_05_2008_c1
Cisco Public
87
Top-of-Rack Consolidated I/O

I/O Consolidation at Access
LAN Core SAN-A 8 8 SAN-B
Distribution
MDS 9500 Cisco Catalyst 6509
STP BLK
POD 1
POD N
Access
Nexus 5000
Adapter: CNA Converged Network Adapter 10 GE/FCoE
Server Cabinet Pair 1

BRKSAN-2821 14572_05_2008_c1
Server Cabinet Pair N

Cisco Public

88
44

Ethernet Host Virtualizer
LAN Core SAN-A 8 8 SAN-B
Distribution
Ethernet Host Virtualizer Active/Active POD 1 POD N
Access
Nexus 5000

BRKSAN-2821 14572_05_2008_c1

Cisco Public

89

VSS Support at Aggregation
LAN Core SAN-A 8 SAN-B
VSS
8
Distribution
VSS Support at Aggregation POD 1 POD N
Access
Nexus 5000

BRKSAN-2821 14572_05_2008_c1

Cisco Public

90
45
Physical Topology40 Servers, ToR

LAN SAN-A SAN-B
4x10 GE Ports
4x10 GE Ports
6x4 GFC
6x4 GFC
Nexus 5000 40x10 GE Ports per Switch
BRKSAN-2821 14572_05_2008_c1
Cisco Public
91
Physical Topology200 Servers

40 4GFC
LAN SAN-A SAN-B
4x10 GE Ports
6x4 GFC
BRKSAN-2821 14572_05_2008_c1
Cisco Public
92
46
Blade Servers with Copper Pass-Through

DC Core MDS SAN-Core 8 MDS SAN-Core
Distribution
POD 1
POD N
Access
Nexus Family
Blade Server
Copper Pass-Through
Blade Server 1
BRKSAN-2821 14572_05_2008_c1
Blade Server N
Cisco Public
Blade Server 1
Blade Server N
93
Blade Servers with Ethernet Switches

DC Core MDS SAN-Core 8 MDS SAN-Core
Distribution
POD 1
POD N
Access
Nexus Family
Blade Server
Ethernet-Only Switch
Blade Server 1
BRKSAN-2821 14572_05_2008_c1
Blade Server 1
Cisco Public
Blade Server 1
Blade Server 1
94
47
Physical Topology 20 Blade Server, ToR

LAN SAN-A SAN-B
2x10 GE Ports
2x10 GE Ports
4x4 GFC
4x4 GFC
Nexus 5000
BRKSAN-2821 14572_05_2008_c1
Cisco Public
95
Topology100 Blade Servers, ToR

LAN SAN-A SAN-B
4x10 GE Ports 8x4 GFC 4x10 GE Ports 8x4 GFC
BRKSAN-2821 14572_05_2008_c1
Cisco Public
96
48
Access Nexus 5000 Distribution Nexus 7000

Distribution
8 MDS 9500 Nexus 7000
Layer 2 Multipath POD 1 POD N
Access
Nexus 5000

BRKSAN-2821 14572_05_2008_c1

Cisco Public

97
Consolidation in the Distribution Layer

Distribution
8 MDS 9500 Nexus 7000
Layer 2 Multipath POD 1 POD N
Access
Nexus 5000

BRKSAN-2821 14572_05_2008_c1

Cisco Public

98
49
Conclusions
BRKSAN-2821 14572_05_2008_c1
Cisco Public
99
Challenges
FCoE redefines consolidated scenarios
Ethernet switch manufacturers will try to enter the FC switching market FC switch manufacturers will try to enter the Ethernet switching market HBA manufacturers will try to enter the NIC market NIC manufacturers will try to enter the HBA market
Deep integration with virtualization will take some time
BRKSAN-2821 14572_05_2008_c1
Cisco Public
100
50
Web Pointers
PCI Express
http://en.wikipedia.org/wiki/Pci_express
IEEE 802.3
http://standards.ieee.org/getieee802/802.3.html
Improvements to Ethernet
http://www.nuovasystems.com/EthernetEnhancements-Final.pdf
IEEE 802.1 activities

http://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf http://www.ieee802.org/1/files/public/docs2007/new-cm-pelissier-enabling-block-storage-0705-v01.pdf http://www.ieee802.org/1/files/public/docs2007/au-ko-fabric-convergence-0507.pdf http://www.ieee802.org/1/pages/802.1au.html http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf
FCoE
http://www.fcoe.com/ http://www.t11.org/ http://www.open-fcoe.org/ http://www.fibrechannel.org/OVERVIEW/FCIA_SNW_FCoE_WP_Final.pdf
TRILL
http://www.ietf.org/html.charters/trill-charter.html
101
Thank You
BRKSAN-2821 14572_05_2008_c1
Cisco Public
102
51
Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books
Available Onsite at the Cisco Company Store

103
Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Dont forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.
BRKSAN-2821 14572_05_2008_c1
Cisco Public
104
52
BRKSAN-2821 14572_05_2008_c1
Cisco Public
105
53

Brksan 2821

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brksan 2821

Uploaded by

Copyright:

Available Formats

BRKSAN-2821 14572_05_2008_c1

2008 Cisco Systems, Inc. All rights reserved.

I/O Consolidation in the Data Center

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

2008 Cisco Systems, Inc. All rights reserved.

Section 1 What Is I/O Consolidation

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

What Is I/O Consolidation

IPC: Inter-Process Communication

I/O Consolidation in the Network

IPC: Inter-Process Communication

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

I/O Consolidation in the Host

All Traffic Goes over 10 GE

2008 Cisco Systems, Inc. All rights reserved.

Cabling and I/O Consolidation

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

I/O Consolidation: Benefits to Customers

Enet Traffic Enet Traffic

Fewer CNAs and Cables

Storage Keeps the Same Management Model as Native FC

Display FCoE Adapter

Less Power and Cooling

2008 Cisco Systems, Inc. All rights reserved.

Merging the Requirements

Doesnt care of the underlying network, provided that:

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Why Has It Not Succeeded Yet?

PCIe 2.0 (aka PCIe Gen 2)

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

CNAs will all be dual-ported for HA

2008 Cisco Systems, Inc. All rights reserved.

Evolution of Ethernet Physical Media

Technology SFP+ CU Copper SFP+ USR

Cable Twinax MM OM2 MM OM3 MM 62.5 m MM 50 m

Distance 10 m 10 m 100 m 82 m 300 m

Power (Each Side) 0.1W

Transceiver Latency (Link) 0.1 s

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Twin-ax Copper Cable

Low error rate (1017)

Application Server Application Server

Application Server Application Server Application Server Application Server

Thinner cable with higher bend radius

Drivers for 10GE to the Servers

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Section 1 Enabling Technologies

2008 Cisco Systems, Inc. All rights reserved.

Three Challenges + One

2008 Cisco Systems, Inc. All rights reserved.

2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Why Are Frames Lost?

It must be dealt with in the bridge/switch

2008 Cisco Systems, Inc. All rights reserved.

Can Ethernet Be Lossless?

Defined in IEEE 802.3Annex 31B

Ethernet PAUSE transforms Ethernet into a lossless fabric