01 Converged Networking Overview

Module 1: Converged Networking Overview

Upon successful completion of this module, you should be
able to:
– Identify the benefits of I/O consolidation in the data center
– Describe what FCoE is and list two new hardware components that
make it possible
– List the major enhancements that must be made to Ethernet to
support FCoE
– Summarize expected topologies for early implementations
© 2008 EMC Corporation. All rights reserved. Module 1: Converged Networking Overview - 1
After successfully completing this module, you should be able to:

 Identify the benefits of I/O consolidation in the data center
 Describe what FCoE is and list two new hardware components that make it possible
 List the major enhancements that must be made to Ethernet to support FCoE
 Summarize expected topologies for early implementations
Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 1

Lesson 1: Why Consolidate I/O?

Upon successful completion of this lesson, you should be
able to:
– Explain the requirements in the data center that make the FCoE
protocol necessary
– Explain what converged networking is and list two benefits.
– List previous barriers to converged networking
After successfully completing this lesson, you should be able to:

 Explain the requirements in the data center that make the FCoE protocol necessary
 Explain what converged networking is and list two benefits.
 List previous barriers to converged networking

Current Data Center Environment
LAN SAN A SAN B
Access
Layer
Infiniband
Ethernet
Fibre Channel
High Performance
Computing
High Performance Data Center
Computing Servers
Notes on terminology that may be new:

HPC = High Performance Computing (Infiniband is a high performance protocol used for
memory to memory transfers.) Requirements: high bandwidth, low latency. The underlying
network protocol to support HPC can be anything as long as these requirements are met (it
doesn’t have to be Infiniband).
IPC = Inter-processor Communications (used for clustering)
HCA = Host Channel Adaptor (used for Infiniband)

Server Requirements
 Parallel LAN SAN infrastructure
 Inefficient use of the network
FC HBA infrastructure
FC HBA  5+ connections per server and higher
adapter and cabling costs:
GE ─ Adds downstream port costs;
CapEx and OpEx.
GE ─ Each connection adds additional points of
failure in the fabric
GE
GE
 Longer lead time for server
provisioning
10/100  Multiple fault domains: Complex
diagnostics
 Management complexity: Firmware,
driver patching, versioning
Requirements:
LAN/IP:
 Ethernet predominant
 Losing some frames OK for TCP
Storage:
 Fibre Channel model predominant
 Losing frames not an option
IPC:
 Any network possible, as long as:
− Low cost
− Low latency
− APIs such as OFED, RDS, MPI, sockets (native or SDP) supported

Converged Networking – The Vision
FC (SAN) 10 GE Pipe
FC (SAN)
FC (SAN) Ethernet (LAN)
Ethernet (IPC)
Ethernet (LAN)
Ethernet (LAN) 10 GE Pipe

FC (SAN)
HCA Infiniband (IPC) Ethernet (LAN)
Ethernet (IPC)
HCA Infiniband (IPC)
CNA = Converged Network Adapter
The vision of converged networking is to take Fibre Channel, Ethernet and IPC traffic, and
merge them onto a single interface, thus saving on the number of required cables and adaptors
per server.

This Solution is GREEN
Converged networking is a “green” technology as cable counts and adapters can be reduced by
approximately 50%, as shown in this example. This not only reduces cable cost (installation and
management complexity), but may reduce the power requirements of data center servers. (Note:
power savings may or may not be significant depending on the adapters and transceivers used)
Note: Copper is a low cost solution for in-cabinet cabling. Then optic can be used to get from
the top of the rack to the core switches. More on cable types is discussed in Module 7: CNA
Configuration and Management.

Barriers to I/O Consolidation

Attempts in the past to consolidate I/O failed because:
 Standard Ethernet is a “Lossy” protocol
– “Best effort” service
 2 Gb or 4 Gb FC vs. GigE
 iSCSI requires different management
 Infiniband – disrupts I/O stack
 Older implementations of the PCI bus were too slow
Some reasons that Converged networking has not been implemented in the past are:
 Ethernet is a Lossy protocol
− Expect dropped frames with recovery at a higher protocol level
− SCSI protocol doesn’t recover from dropped frames in a very timely fashion
 2 Gb or 4 Gb FC vs. GigE
− GigE just didn’t have the bandwidth to carry heavy FC traffic
 iSCSI requires different management
− Zoning: IQN vs. WWN
− Relies on TCP/IP
 Infiniband – disrupts I/O stack (from operating system middleware to physical media)
 Old versions of the PCI bus were too slow
Other protocols that you may be aware of, were never meant to be I/O consolidation protocols:
 FCIP is a FC tunneling protocol
− Relies on TCP and GigE
− Point solution best used for extending FC distance
 iFCP is a routing (gateway to gateway)
− Point solution best used to route between fabrics. Relies on NAT

PCI Express
 PCI Express (PCIe): Computer expansion card
interface format designed to replace PCI, PCI-X, and
AGP
 PCI Express 1.1: Full duplex serial links at 2.5 Gb/s
(2 Gb/s at datalink):
– Supports speeds from 2 Gb/s (x1) to 32 Gb/s (x16)
– x8 required for 10 Gigabit Ethernet
– Product shipping
 PCI Express 2.0 (PCI Express Gen 2)—Doubles

bandwidth per serial link from 2 Gb/s to 4 Gb/s:
– Spec available since January 2007
– No CNA products currently shipping
The currently shipping PCI standard (PCIe) supports much higher bandwidths than in the past.
PCIe with x8 bus speed handles 16Gb/sec.
Note: dual ported PCIe x8 CNAs can’t run 10 Gbps line rate on both ports simultaneously.

I/O Consolidation with FCoE

Summary of Customer Benefits:
 Fewer host adapters and cables

 Same management model as Fibre Channel
 Not a “Gateway” protocol
 Saves $$$
I/O consolidation offers all of these benefits:

 Fewer host adapters and cables
 Same management model as Fibre Channel
 Not a “Gateway” protocol

Lesson 2: FCoE Technology

able to:
– Describe FCoE
– Identify two new hardware components that make it possible

FCoE – Fibre Channel Frame Encapsulation
Normal Ethernet frame, Ethertype = FCoE
Same as a physical Fibre Channel frame

Ethernet
Header
Header
Header
FCoE
CRC
EOF
FCS
FC
FC Payload
Control information: Version, ordered sets (SOF, EOF)
FCoE is a new T11 standard being developed by the FC-BB-5 working group. Since there is a
very large install base of Fibre Channel based SANs, a new standard was needed that is
compatible with the install base, but still meets the goals and provides the benefits of converged
networking. FCoE is the result of this collaboration.
FCoE is accomplished by simply placing the standard Fibre Channel frames, without
modification, into Ethernet frames. A short FCoE header and trailer is added that has the Fibre
Channel SOF and EOF delimiters included. Then the whole FCoE frame is encapsulated into a
standard Ethernet frame. This frame encapsulation and de-encapsulation can be done by
hardware (at basically the line rate), so there is little added latency.
FCoE does have some strict Ethernet requirements that must be met:
 10 Gb/s Ethernet (This is the minimum bandwidth required for converged networking)
 Ethernet jumbo frames:
− Max Fibre Channel frame has a 2112 byte payload. This doesn’t fit into a standard Ethernet 1500 MTU,
and we don’t want to fragment the FC frames. FCoE requires an MTU of 2500. This is sometimes
referred to as a “baby jumbo” frame. (Note that regular Ethernet traffic on the NEX-5020 will still use an
MTU of 1500 as the default)
 Lossless Ethernet:
− Matches the lossless behavior of Fibre Channel. This is necessary because SCSI is optimized for the error
free condition. The simplified SCSI recovery stack assumes errors will be very rare. The tradeoff made in
SCSI for high performance is lengthy recovery times for dropped packets.
− More on this to follow

New Hardware Needed for FCoE
CNA (Emulex, QLogic)
FCoE Enabled 10 Gb Ethernet switch

Cisco NEX-5020
The new CNAs from Emulex and QLogic, and the new NEX-5020 switch all support 10 Gig
Lossless Ethernet, and do FCoE frame encapsulation de-encapsulation in hardware. The NEX-
5020 switch supports both lossless 10 Gig Ethernet, and 1/2/4 Gbps Fibre channel.

Lesson 3: Enhanced Ethernet

able to:
– Define the term “Enhanced Ethernet”
– Summarize the features required in DCE / CEE

 Define the term “Enhanced Ethernet”
 Summarize the features required in DCE / CEE

Converged Enhanced Ethernet (CEE)
FCoE requires the following Ethernet Extensions:

– Lossless Ethernet
 Priority-based Flow Control (PFC)
– Bandwidth Management
 Enhanced Transmission Selection (ETS)
– Advertise support for enhanced Ethernet features

 Data Center Bridging Exchange (DCBX) Protocol
As was mentioned earlier, one of the reasons converged networking has failed in the past is that
sending SCSI information over a lossy network like Ethernet has some performance problems.
To fix this, enhancements are needed for Ethernet. To support FCoE, a group of enhancements
is being developed under the collective designation: “CEE” (Converged Enhanced Ethernet).
CEE (pronounced “see”) includes the three extensions shown here.
Enhanced Ethernet is a collection of IEEE-based enhancements to classical Ethernet that
provide end-to-end QoS functions. These include:
 Lossless Ethernet which uses priority groups and virtualizes links and allocates resources per
traffic class (Priority flow control by traffic class)
 Bandwidth Management through the ETS standard
CEE does not disrupt the existing LAN/SAN infrastructure. There is an advertising feature
(DCBX) that allows nodes at either end of a link to determine if CEE features are supported.
Benefits of Ethernet enhancements:
 Eliminates transient and persistent congestion
 Lossless fabric: No-drop storage links
 Deterministic latency for HPC clusters
 Enables a converged Ethernet fabric for reduced cost and complexity

What’s in a Name?
Three different names for Enhanced Ethernet:
Acronym Definition
DCB Data Center Bridging (IEEE)
CEE Converged Enhanced Ethernet (IBM, Brocade)
DCE Data Center Ethernet (Cisco Trademark)
 Cisco DCE includes CEE, plus the following:

– Layer 2 Multipath (shortest path bridging)
– Lossless fabric (PFC + intelligent switch fabric arbitration mechanism)
– Congestion notification (end-to-end congestion management and notification)
The following Question and Answer is from Cisco and may shed more light on this subject:
Question: Why are there different names for the collection of these Ethernet enhancements
(DCE™, CEE, and DCB) and how do the terms differ?
Answer: Different organizations created different names to identify the collection of the
specifications, all based on the same core specifications. IEEE has used the term "Data Center
Bridging (DCB)." IEEE typically calls a standard specification by a number: for example, IEEE
802.1az. IEEE did not have a way to identify the group of specifications with a standard
number, so the organization grouped the specifications into DCB.
The term "Converged Enhanced Ethernet (CEE)" was coined by IBM, again to reflect the core
group of specifications, to gain consensus among industry vendors (including Cisco) as to what
a Version 0 list of the specifications would be before they all become standards.
Cisco uses the term "Data Center Ethernet" to refer to its architecture for a next generation of
Ethernet for the data center. Cisco Data Center Ethernet is a superset of the CEE and DCB
proposals, using the same three specifications: PFC, ETS, and DCBX. In addition, Cisco
includes L2MP, lossless fabric, and congestion notification. The DCE™ designation will be
used when a Cisco product or Cisco partner product includes those specifications and meets the
requirements as defined by the Cisco Data Center Ethernet architecture.

Lesson 4: FCoE Implementations

able to:
– Summarize expected topologies for early implementations

•Summarize expected topologies for early implementations

Expected First Implementation of FCoE

 Non-production (test) environments
 Use FCoE to add new servers to edge of existing LANs
and SANs

Top of Rack Solution
SAN A LAN Core SAN B
Distribution Layer:
4 Fibre Channel Switch
Catalyst 6509
1/2/4 Gb Fibre 10 GE
Channel
Access Layer:
NEX-5020
Ethernet
Server Rack Fibre Channel
Expected First Implementation of FCoE

 Non-production (test) environments
 Use FCoE to add new servers to edge of existing LANs and SANs
19
End-of-Row I/O Consolidation
Core
To SAN
Core
To SAN
LAN Core
Distribution;
8 Cisco Nexus 7000
FC Switch: MDS or Brocade
8
4 x 10 GE
4 x 4 G FC Server Pod
Aggregation:
Pod 1 Pod N FC Director
Nexus 7000
Access:
NEX-5020
Server Rack N Server Rack N 10 GE/FCoE

Server Rack 1 Server Rack 1
The Fibre Channel Switches in this example could be implemented with MDS-Series or B-
Series switches.

4Q2008 Supported FCoE Configurations
* * * Always check the EMC support matrix for the latest supported configurations * * *
Configurations supported in Q4 '08 include Emulex and QLogic CNAs, IBM Servers and
Cisco/Nuova switches. Operating systems supported include Red Hat 5.2, SLES 10 SP2, and
VMware 3.5. RPQs may be submitted for other servers and operating systems. Please consult
the ESM for more specific information on supported configurations.
Note: Celera support is not for FCoE, but for NAS and iSCSI over 10 GigE. Celera does not
qualify NIC's nor Ethernet switches. However the CNA's are using Intel's very widely used
Oplin ASIC and driver stack. Also the NEX-5020 passed UNH's Ethernet IOL tests.

Module Summary
Key points covered in this module:
 Converged Networking will save $$ with less cabling and
fewer adapters to manage.
 FCoE is a converged networking protocol that
encapsulates standard Fibre Channel Frames into
Ethernet Frames. New CNAs and FCoE switches make
this possible.
 FCoE requires 10 GigE, Jumbo Frames and Lossless
Ethernet (CEE).
 First implementations are expected to be in test
environments, and at the edge of existing SANs / LANs.
These are the key points covered in this module.

•Converged Networking will save $$ with less cabling and fewer adapters to manage.
•FCoE is a converged networking protocol that encapsulates standard Fibre Channel Frames into
Ethernet Frames. New CNAs and FCoE switches make this possible.
•FCoE requires 10 GigE, Jumbo Frames and Lossless Ethernet (CEE).
•First implementations are expected to be in test environments, and at the edge of existing SANs
/ LANs.

Learning Exercise: Check Your Knowledge
1. What are two benefits of converged

networking?
2. What is FCoE?
• What two new hardware components are
necessary?
3. What are the major enhancement that
must be made to Ethernet, to support
FCoE?
4. What Topology is expected for early
implementations of FCoE technology?

Learning Exercise: Answers
1. What are two benefits of converged

networking?
2. What is FCoE?
• What two new hardware components are
necessary?
3. What are the major enhancement that
must be made to Ethernet, to support
FCoE?
4. What Topology is expected for early
implementations of FCoE technology?
Answers:
1. Two benefits of Converged networking using FCoE are fewer cables and adapters, less cost
and maintaining FC Management model.
2. CNAs and FCoE enabled switches are two hardware components necessary for FCoE.
3. Ethernet enhancements required for FCoE are Lossless Ethernet, bandwidth management, and
Data Center Bridging Exchange protocol (DCBX)
4. Early implementations expect data consolidation for the first few meters from the server; at
the edge of the LAN SAN.

01 Converged Networking Overview

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 Converged Networking Overview

Uploaded by

Copyright:

Available Formats

Module 1: Converged Networking Overview

Module 1: Converged Networking Overview

After successfully completing this module, you should be able to:

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 1

Lesson 1: Why Consolidate I/O?

After successfully completing this lesson, you should be able to:

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 2

Current Data Center Environment

LAN SAN A SAN B

Notes on terminology that may be new:

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 3

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 4

Converged Networking – The Vision

Ethernet (LAN) 10 GE Pipe

CNA = Converged Network Adapter

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 6

This Solution is GREEN

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 7

Barriers to I/O Consolidation

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 8

 PCI Express 2.0 (PCI Express Gen 2)—Doubles

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 9

I/O Consolidation with FCoE

 Fewer host adapters and cables

I/O consolidation offers all of these benefits:

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 10

Lesson 2: FCoE Technology

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 11

FCoE – Fibre Channel Frame Encapsulation

Normal Ethernet frame, Ethertype = FCoE

Same as a physical Fibre Channel frame

Control information: Version, ordered sets (SOF, EOF)

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 12

New Hardware Needed for FCoE

CNA (Emulex, QLogic)

FCoE Enabled 10 Gb Ethernet switch

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 13

Lesson 3: Enhanced Ethernet

After successfully completing this lesson, you should be able to:

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 14

Converged Enhanced Ethernet (CEE)

FCoE requires the following Ethernet Extensions:

– Advertise support for enhanced Ethernet features

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 15

CEE Converged Enhanced Ethernet (IBM, Brocade)

DCE Data Center Ethernet (Cisco Trademark)

 Cisco DCE includes CEE, plus the following:

– Congestion notification (end-to-end congestion management and notification)

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 16

Lesson 4: FCoE Implementations

After successfully completing this lesson, you should be able to:

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 17

Expected First Implementation of FCoE

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 18

Top of Rack Solution

SAN A LAN Core SAN B

Expected First Implementation of FCoE

End-of-Row I/O Consolidation

Server Rack N Server Rack N 10 GE/FCoE

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 20

4Q2008 Supported FCoE Configurations

Copyright © 2008 EMC Corporation. Do not Copy - All Rights Reserved. - 21

These are the key points covered in this module.