1

High Speed WAN:

Traditional and emerging
technologies
Emerson Moura, Distinguished Systems Engineer
BRKOPT-2117

CCIE# 15356
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
Introduction
• Business requirements are driving higher speeds in the WAN.
• Higher speeds can lead to higher cost and more complexity, pushing for
technology innovations and new service paradigms.
• There are many options to migrate to higher speeds in the WAN, each one has
it’s pros and cons.
• Goal is to understand and minimize the impact of the tradeoffs.
Challenges in the WAN
Bandwidth
Requirements

Budget

Expected
$/Bandwidth

time
Tackling the bandwidth challenge
• New higher-speed interfaces
• Advances in optical / photonics
• Leverage economies of scale
• Packet switching based services
• Cost reductions
• Changes in network architectures
• Technology Convergence
• WAN Optimization
 40G, 100G, 200G and Beyond
 Silicon Photonics, Modulations
 Ethernet
 IP/MPLS, MEF Services
 Moore’s Law, new technologies
 CDNs, SDNs, Distributed DCs
 Ethernet and Transport
 TCP Optimization, Acceleration
Finding the right balance = Engineering

Faster Efficient
Cheaper Trade-Offs Reliable
Simpler Manageable
Constraints Requirements
Expected Technical Attributes of WAN Services

Low Overhead/Efficiency

Robustness/OAM/Protection

Interoperability/Transparency

Flexibility/Simplicity

Scalability/High Speeds
 WAN Transport Technology Map
Note: Partial, Simplified View. Only References to Fiber Optics Interfaces Were Included.

Native Ethernet Ethernet + G.709 POS

Non-IP IP
802.3 RFC 1662

802.1ah
RFC 2615
802.1Q / 802.1ad

802.3ae WWDM
G.707
802.3ae WAN-PHY G.709
GR-253-CORE
802.3ae LAN-PHY

802.3ae
G.957
802.3aq
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
Traditional High-Speed* WAN Technologies

• POS – Packet over SONET/SDH, commonly found in Service Providers and in

large Enterprises.
• Classic Ethernet – 802.3ae LAN/WAN, widely used in Enterprise switches and
routers without carrier grade protocols.

* Transmission rates above DS3/E3

Packet over SONET/SDH
Packet Over SONET/SDH (PoS)
• PPP/HDLC over SONET/SDH frames. SONET/SDH Hierarchy
Very efficient with low overhead.
SONET SDH Bit Rate
• Standards : (Mbps)
• RFC 1662  PPP in HDLC-like Framing
OC-3 STM-1 155.52
• RFC 2615  PPP over SONET/SDH
• G.707/GR-253-CORE  Framing format, OC-12 STM-4 622.08
hierarchy, etc
OC-48 STM-16 2,488.32
• Scales from 155Mbps up to 40Gbps.
OC-192 STM-64 9,953.28
• Available in Channelized interface options
with very good granularity (down to DS0). OC-768 STM-256 39,813.12
• Excellent OAM and protection - industry SONET/SDH Limit – 40Gbps
benchmark.
• Most efficient technology for applications
with small packet sizes (eg. VoIP).
Pros and Cons of PoS Technology
Pros Cons
• Fault Management • Cost
• Performance Management • Single class of service
• Fast, reliable protection • Fixed rates with rigid hierarchy
• Overhead Efficiency • Port density
• Full interoperability with legacy • Stopped at 40Gbps
transport
Calculating PoS Data Rate for OC-192/STM-64
STS-192/VC-4-64c Frame Period = 125us (8kfps)

Efficient HDLC encapsulation

Frame Structure: 270 Columns x 9 Rows

• 9 Columns Fixed Overhead per Frame, 1 Column Path-Overhead per Frame
• 64 Concatenated Frames in OC-192/STM-64

 Line Rate: (270 x 9)Bytes x 8 Bit/Byte x 8kfps x 64 = 9.953280 Gbps

 Payload: ((270-9-1) x 9)Bytes x 8 Bit/Byte x 8kfps x 64 = 9.58464 Gbps

PoS Efficiency for IP Traffic at OC-192/STM-64
Maximum Theoretical Values – IMIX Traffic Mix (IP)
IP Packet Packet Byte IP Traffic
Size (Bytes) % % (Mbit/s)
40 58.7 5.9 555.9
44 2.0 0.2 20.9
576 23.66 34.5 3,227.7
1500 15.67 59.5 5,567,1
Total (Mbps) 9,371.5
Line Rate Efficiency 9,371.5 / 9,953.3 = 94.2%
Payload Efficiency 9,371.5 / 9,584.6 = 97.8%

POS encapsulation is very efficient (low overhead)

 PoS OAM Architecture
OC-N/STM-N SONET/SDH OC-N/STM-N

SONET/SDH Frame Network SONET/SDH Frame

Section Section Section

Line
Path

Client OAM (PDH/SONET/SDH)

• Several OAM Layers are provided: Client, Section, Line, Path. (end-to-end visibility)
• Comprehensive set of features: PM, FM, Signaling, Tracing, Protection triggering.
• OAM information is carried and processed in every SONET/SDH frame (near real-time).
• Seamless OAM integration between layers.
Classic Ethernet
 Ethernet Frames
802.3 Frame
Bytes > 12 7 1 6 6 2 46-1500 4
S
Type/ 802.2 Header
Field > IFG Preamble O DA SA FCS
Length and Payload
F
Service Frame*
Packet
Worst case Overhead = 38 Bytes

802.1Q Frame
12 7 1 6 6 4 2 46-1500 4
Bytes >
S
Type/ 802.2 Header
IFG Preamble O DA SA TAG FCS
Field > Length and Payload
F
Service Frame*
Packet
Worst case Overhead = 42 Bytes

* MEF 10.3 Definition of Service Frame

Classic Ethernet Pros and Cons
Pros Cons
• Cost • No end-to-end PM
• Flexible and granular data rates • No end-to-end FM
• Multiple Classes of Services • Poor protection mechanisms
• Multiple connectivity options • Loops
• Speeds up to 100Gbps • Potential interoperability issues with
legacy transport
• Ubiquitous technology
• Large overhead
 Example: 10 Gigabit Ethernet Layer Diagram*

Media Access Control (MAC)

Full Duplex

10 Gigabit Media Independent Interface (XGMII) or

10 Gigabit Attachment Unit Interface (XAUI)
Data Rate 10 Gbps 10 Gbps 9.29 Gbps

WWDM Serial Serial

LAN PHY LAN PHY WAN PHY
(8B/10B) (64B/66B) (64B/66B + WIS)
Interface Rate 12.5 Gbps 10.31 Gbps 9.95 Gbps
WWDM Serial Serial Serial Serial Serial Serial
PMD PMD PMD PMD PMD PMD PMD
1310 nm 850 nm 1310 nm 1550 nm 850 nm 1310 nm 1550 nm

LX4 SR LR ER SW LW EW
* Fiber options shown only. Does not include 10GBASE-CX4, 10GBASE-T
Calculating 802.3ae WIS (WAN-PHY) Data Rate
Frame Period = 125us (8kfps)

STS-192/VC-4-64c

 As previously described, OC-192/STM-64 Payload = 9.58464 Gbps

 MAC-layer data rate (include line coding): Data-Rate x 64/66 = 9.294196 Gbps
Classic Ethernet Efficiency for IP Traffic at 10GE
Maximum Theoretical Values – IMIX Traffic Mix (IP)
LAN-PHY WAN-PHY

Packet Size Packet Byte % IP Traffic (Mbit/s) IP Traffic (Mbit/s) IP Traffic (Mbit/s) IP Traffic (Mbit/s)
(Bytes) % 802.3 Frames 802.1Q Frames 802.3 Frames 802.1Q Frames
40 58.7 5.9 541.1 536.2 502.9 498.3
44 2.0 0.2 20.2 20.1 18.9 18.7
576 23.66 34.5 3,142.4 3,113.7 2,920.7 2,893.9
1500 15.67 59.5 5,419.9 5,370.4 5,037.4 4,991.3
Total (Mbps) 9,123.8 9,040.4 8,479.8 8,402.3
Line Rate Efficiency 9,123.8 / 10,312.5 = 88.5% 9,030.4 / 10,312.5 = 87.6% 8,479.8 / 9,953.3 = 85.2% 8,402.3 / 9,953.3 = 84.4%

Data Rate Efficiency 9,123.8 / 10,000.0 = 91.2% 9,030.4 / 10,000.0 = 90.4% 8,479.8 / 9,584.6 = 88.5% 8,402.3 / 9,584.6 = 87.6%

Ethernet has a high encapsulation tax for small packets

802.3ae WIS (WAN-PHY) Framing Considerations
• Uses SONET/SDH frame with minimum overhead functions
• Requires VC-4-64c (64 contiguous VC-4s) or
STS-192c (192 contiguous STS-1s)
• Clock tolerance differs from ITU-T *:
• 802.3ae: +/- 20ppm
• ITU-T: +/- 4.6 ppm
• Result: generates higher pointer justifications in SONET/SDH networks.

• There is no WAN-PHY or equivalent for 40G or 100G.

Better Applicable to xWDM Transport

* Some component providers claim better specifications than IEEE.
 802.3ae WIS OAM Overhead Reference

Regeneration Section
Path Overhead Fixed Stuff (63 Columns)

Section Overhead
1 2 3 4 5 6 7 8 9

1 A1 A1 A1 A2 A2 A2 J0 Z0 Z0 J1 ….

2 B1 E1 F1 B3 ….
3 D1 D2 D3 C2 ….

Pointers 4 H1 H1 H1 H2 H2 H2 H3 H3 H3 G1 ….

5 B2 K1 K2 F2 ….
6
Multiplex Section

D4 D5 D6 H4 ….
Line Overhead

7 D7 D8 D9 F3/Z3 ….

8 D1 D1 D1
K3/Z4 ….
0 1 2
9 S1 M1 N1/Z5 ….

Undefined Overhead Bytes – SONET/SDH Defined Bytes Not Used

Set to Zero by 10GE WAN-PHY– Set to Zero

XX Fixed Value Bytes XX Provisioned Value Bytes

XX Calculated Value Bytes

OAM in Native Ethernet Networks
Ethernet Ethernet Ethernet
Network
FCS FCS FCS FCS FCS

Upper Layers Fault Detection Mechanisms (ex. BFD, IGP Timers, etc.)

• Up to 10GE, traditional Ethernet provides simplest OAM possible – CRC/FCS.

• Lacked tools for PM, FM, Signaling, Tracing, Protection, etc.
• Physical and logical failures have to be handled by upper layer protocols timers (BFD, IGPs, STP, etc).
Ethernet and Transport OAM Interoperability

Ethernet
SONET/SDH Ethernet
Network
FCS Section Section Section FCS
Line
Path

Upper Layers Fault Detection Mechanisms (ex. BFD, IGP Timers, etc.)

• OAM is provided only at the Transport layer.

• No OAM Interworking between layers. Typical solution – Failure detected at transport layer triggers
client port squelching.
• Network failures or degradation requires upper layer protocols for detection and convergence.
• Dribbling conditions can’t be easily detected.
40GE and 100GE
• IEEE 802.3ba added 40Gb/s and 100Gb/s Ethernet interfaces.
• Not just higher speeds – brings different interface architectures.
• Standard ratified June 17, 2010.
• Aligned with ITU Study Group 15, Next Generation Optical and Transport
Networks (OTN):
• ODU4 defined and completed, compatible with 100GE
 802.3ba Interfaces
Interface PHY Layer Link Distance
40GBASE-CR4 4 Lanes of Shielded 7m/23ft
Copper Bal. Cabling
40GBASE-SR4 4 Lanes of Multimode 100m/330ft
Fiber
40GBASE-LR4 4 WDM Lanes on 10km/6mi IEEE Interfaces
Singlemode Fiber Reach up to
100GBASE-CR10 10 Lanes of Shielded 7m/23ft 40km/25 Miles
Copper Bal. Cabling

100GBASE-SR10 10 Lanes of Multimode 100m/330ft

Requires OTN/DWDM
Fiber for Extended Reaches

100GBASE-LR4 4 WDM Lanes on 10km/6mi

Singlemode Fiber
100GBASE-ER4 4 WDM Lanes on 40km/25mi
Singlemode Fiber
Source: 802.3ba – Table 80-1
802.3ba 100GE Specifications Reference

WDM
Not compatible
with ITU-T G.694

Source:
IEEE 802.3ba
Table 88-8
PHY Level Error Monitoring for Ethernet
Ethernet PCS BIP
OTN BIP
PCS BIP PCS BIP
MON MON

40GE/100GE OTN OTN OTN 40GE/100GE

TxP Network TxP

• BIP-8 monitoring incorporated into 802.3ba standard (40GE/100GE)

• Improved error monitoring, without the need for OTN encapsulation
• Transparent transport over the OTN network
• Proactive monitoring with user configurable SF/SD BER thresholds, etc

Theory Actual BER

Measured
1.00E-09 1.00E-08 1.00E-07 1.00E-06 1.00E-05 1.00E-04 1.00E-03 1.00E-02 1.00E-01 1.00E+00
1.00E+00

1.00E-01

1.00E-02
Invalid
Sync
1.00E-03

Indicated BER
1.00E-04

1.00E-05

BIP-8
1.00E-06

1.00E-07

1.00E-08

1.00E-09
40G/100G Client Interface Options
TxP
40G/100G Router/Switch
???
Native Ethernet
• Transport maps Ethernet to OTN
• Lowest cost .. but perceived lack of OAM
• IEEE 802.3ba added BIP error check
• SONET/OTN like error monitoring
SF/SD BER thresholds, etc
40G/100G DWDM Network
DP-QPSK DP-QPSK
OTU4
OTN “Short Reach” Client
• Router maps Ethernet to OTN
• Good OAM.. but OTN requires serial optics
• ITU added support for parallel optics:
OTL4.4, OTL4.10 OTL3.4, etc
• Can run OTN clients (both 40G and 100G)
over ‘Ethernet’ CFP optics.
OTN/SONET versus Ethernet PHY OAM Reference

OTN/SONET OAM Ethernet Phy OAM Comments

LOS LOS Both based on monitoring optical power level.

PCS_Block_Loss provides a robust, deterministic link failure

LOF/LOMF PCS_Block_Loss
alarm similar to LOF in SONET/OTN.
LF is inserted at the physical layer to indicate to the
AIS LF (Local Fault) downstream MAC of an upstream failure. Same function as AIS
in SONET/OTN.
When a local MAC receives a LF condition, it transmits a RF
BDI RF (Remote Fault) condition to the remote MAC. Same function as RDI/BDI in
SONET/OTN
Monitors errors in 66bit block sync headers. Can be used to
64/66 Sync Header Errors
determine bit error ratio of the link
Error Monitoring:
OTU, ODU BIP etc Same error checking mechanism as SONET/OTN. Added by
PCS Lane BIP-8 monitor *
802.3ba to improve error monitoring for 40GE/100GE .

Indicates PCS is detecting BER> e-4. Same as OTU-SF-BER,

OTU-SF-BER PCS_high_BER
except fixed and not user programmable.

Could implement a user configurable BER alarm, based on

OTU-SD-BER PCS_SD_BER * monitoring PCS errors. Could also use to trigger LF/RF and
take interface down.

Performance monitoring TCA (threshold crossing alert). Could

TCA-BER TCA_PCS_BER *
implement same thing using PCS error monitoring.
* Added in IEEE 802.3ba for 40GE/100GE
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
Introducing Metro Ethernet Forum CE 2.0 Services
CE 2.0
Services

Point to Point Multipoint Wholesale

E-Line E-LAN E-Tree E-Access

EP EVP EP EVP EP EVP EP EVP

Line Line LAN LAN Tree Tree Access Access

BRKOPT-2117 Reference: BRKSPG-2720 Session

Three Approaches to Ethernet in the WAN
• Use Ethernet as client and traditional TDM technologies as transport;
• Make Ethernet more carrier grade and use it as both client and transport;
• Use Ethernet as client and another carrier grade packet switching technology as
transport (eg. MPLS);

A fourth approach….
• The combination of two or all approaches above!
Ethernet over Traditional Transport Networks
Ethernet

SONET/SDH OTN

VC/STS
Optical Electrical
Containers

OTN
xWDM
Switches

Sub- ODU
Wavelength
wavelength containers

ODU ODU
Containers Containers

Transparent
(2R)
Mapping Ethernet to TDM with GFP

Ethernet MAC Frame GFP Frame

2 PLI

Bytes
2 cHEC
12 IPG 2 Type
7 Preamble 2 tHEC
1 SOF Delimiter 0-60 GFP Ext. Header
6 DA
Bytes

6 SA
2 Length/Type
Data GFP Payload

Data
4 FCS

GFP: Generic Framing Procedure

 Ethernet over SONET/SDH Containers
Contiguous Concatenation

VC-N(1)/STS-N(1)
VC-N-mc / STS-N-mc
m times containers of same size that
VC-N(m)/STS-N(m)
must be contiguous.

Virtual Concatenation (VCAT)

VC-N(1)/STS-N(1)
VC-N(2)/STS-N(2) VC-N-mv / STS-N-mv
unavailable
VC-N(3)/STS-N(3) m times containers of same size that can be
VC-N(m)/STS-N(m) anywhere in the SONET/SDH structure.

Link Capacity Adjustment Scheme (LCAS)

Allows in-service scale up or down the size of the virtual container.
Ethernet over SONET/SDH Pros and Cons
Pros Cons
• Well-known, simple service; • Additional overhead and latency;
• Sub 50ms Protection; • Rigid bandwidth (n x container size);
• Deterministic, no shared • No OAM or signaling integration;
bandwidth*;
• Point-to-point services only **;
• Huge infrastructure deployed and
• No statistical multiplexing **;
usually available;
• No service multiplexing **;
• Native data scrambling;
* Packet networks could also achieve this. • Usually limited bandwidth available.
** Refers to native support on SONET/SDH technology.
Optical Transport Network (OTN)
• Standards : G.709 Hierarchy
• G.709  Hierarchy and frame Frame Payload (OPU)
structures ODU0 1,238,954 kbit/s
• G.872  Architecture OTU1 2,488,320 kbit/s
• G.798  Management functions etc OTU2 9,995,276 kbit/s
OTU3 40,150,519 kbit/s
• Defines G.709 as a framing OTU4 104,355,975 kbit/s
technology and hierarchy that is very
similar to SONET/SDH.
• G.709 started as a wrapper around
WDM client signals to improve reach Payload
and manageability
• Evolved to a complex multiplexing
hierarchy that enables a service
layer
 Reference
Optical Transport Network (OTN) Model
Optical Channel (Och) OTN Options
Optical Multiplex Section (OMSn)* • Optical OTN:
Optical Transmission Section (OTSn)* • DWDM Transport
• All Optical Network
MUX/DEMUX
MUX/DEMUX
Optical Amplifiers
• Lambda or Sub-lambda
services
• Cross-connect or switching at
Optical
Channels the Lambda/Wavelength Level

• Electrical OTN:
• SONET/SDH Evolution
OTS OTS OTS
• Switching of ODU-k containers
OMS • Circuit services
• Can use Optical OTN as
OCh transport
*Under Definition.
 Reference
ITU-T G.709 Mapping
IP

ESCON/FC Ether

GFP-T GFP-F G.Sup43 SONET/SDH ATM

Client Payload

OPUk OH Client Payload

ODUk OH OH Client Payload

OTUk OH OH OH Client Payload OTUk FEC

0 = 1.25G (ODU Only)

1 = 2.5G
k Indicates the Order: 2 = 10G
3 = 40G
4 = 100G
Forward Error Correction (FEC)
Compensates for Optical Impairments
• FEC extends reach and 0
–1
design flexibility, at –2 Raw Channel BER=1.5e-3
“silicon cost” –3
–4
• G.709 standard improves –5
OSNR tolerance by 6.2 dB –6

Log (BER)
(at 10–15 BER) –7
–8

• Offers intrinsic performance –9

–10
monitoring (error statistics) –11 Uncoded
No FEC
–12
• Higher gains (8.4dB) possible by EFEC=8.4 dB
–13
enhanced FEC (with same G.709 –14
FEC=6.2 dB

overhead) –15
4 5 6 7 8 9 10 11 12 13 14 15

FEC/EFEC Extends Reach and Offers 10–15 BER S/N (dB)

Ethernet over G.709 Encapsulation
• G.709 was developed to support SONET/SDH networks, not Ethernet.
• Ethernet line rates and G.709 payloads were not directly compatible.

Ethernet Line Rate Original OTN Payload (ODU-N)

GE: 1.25 Gbps OPU1 = 2.488 Gbps
10GE LAN-PHY: 10.325 Gbps OPU2 = 9.953 Gbps

• Solution to the GE interoperability problem: ODU0 (1.25Gbps payload)

• Different solutions exist to the 10GE interoperability problem (next slides).
• OTN provides full compatibility with 100GE (ODU4).
The Challenge to Transport
10GE LAN-PHY over G.709
Mapping
GFP-F into OPU2
Overclocked OPU2
Rate Adaptive Mapping into
OPU2
GFP-F Mapping into Extended
OPU2
Reduced Rate into OPU2
OTN ODUFlex
Description
Mapping Ethernet Frames Using • Defined in G.7041
GFP-F into OPU2
Mapping of 64B/66B Encoded
Data into OPU2 with Increased
Clock Rate
Mapping of 64B/66B Encoded
Data into OPU2 with Idle
Removal and Optional Flow
Control
Extended OPU2 Payload Using
GFP-F
Lower Ethernet Rate so that
64B/66B Encoded Data Can Be
Mapped into a Standard Rate
OPU2
Split 64B/66B Encoded Data
into 5 OPU1k VC
Advantages
• Bit level Ethernet transparency
• Bit level Ethernet transparency
• Maintains G.709 bit rate
• Information level Ethernet
transparency – preserves
preamble and ordered sets
• Maintains G.709 bit rate
• Maintains G.709 bit rate
• Bit level Ethernet transparency
• Transparency transport of 10GE
LAN @10.3 G
Reference
Reference
Disadvantages
• Not transparent to overhead
information such as preamble or IPG
• Does not support full Ethernet rate
• Requires increasing the G.709 bitrate
• Two different mappings and rates
currently proposed
• Does not support full Ethernet rate
• Requires client to reduce rate by
inserting idles or support of 802.3x flow
control
• No bit level transparency
• No G.709 Compliant (OPU OH used
as Payload)
• Requires defining a new Ethernet
interface with a lower bit rate
• 2 lambdas requested to transport a
single 10GE LAN
Transporting 40GE over OTN
• IEEE has made 40GE more friendly to OTN.
• 40GE LAN – same 64B/6BB coding as 10GE LAN.
• Result – 41.25 Gbps serial signal.
• ODU3 Payload (OPU3) rate: 40.15Gbps
• Solution:
• Transcode 64B/66B into a more efficient 1024B/1027B block code.
• Resulting stream = 40.11Gbit/s – fits into OPU3.
• Map resulting stream in the OPU3 using ITU-T GMP.
100GE over OTN
• ITU-T has optimized the OTU4 rate for 100GE.
• OTU-4 rate is 111.809974 Gbit/s.
• Beneficial for 100GE – full compatibility.
• Challenges for OTN:
• breaks the 4x increase. 100G is not a clear integer of lower ODUk signals
(ex. 4x ODU3 = 160Gbps).
• What’s next? OTU-5 (400Gbps vs 1Tbps)
Additional OTN Containers and Capabilities
• ODU-0 – smaller 1.25Gbps container.
• ODU-3e – similar to ODU-2e, for 40GE and 4x10GE.
• ODU-4 – compatible with 100GE.
• ODU-Flex – custom ODU rates (nx ODU-0/1.25Gbps).
• GMP – Generic Mapping Procedure.
Ethernet over OTN Pros and Cons
Pros
• Similar to SONET/SDH*; plus
• More bandwidth available compared
to SONET/SDH.
* See slide 45.
Cons
• Similar to SONET/SDH*; plus
• May not be transparent to 10GE or
40GE (bit level transparency).
• Big containers only, not suitable for
lower speed services.
 Making Ethernet Carrier Grade

New standards complement Ethernet to address carrier grade,

high-performance MAN/WAN environments.

Transport
Scale OAM Resiliency Synchronization *
Encapsulation

802.1ad Y.1731 G.8031 1588v2 OTN

802.1ah 802.1ag G.8032 SyncE G.709
802.3ah
*Not covered in this session.
 Ethernet Frame Formats
Addressing Service Provider Scale Needs
QinQ or 802.1ad Frame (PB)
12 7 1 6 6 4 4 2 46-1500 4
S
S C Type/ 802.2 Header
IFG Preamble O DA SA FCS
TAG TAG Length and Payload
F
Overhead = 46 Bytes

802.1ah Frame (PBB)

Bytes > 12 7 1 6 6 4 2 4 6 6 4 4 2 46-1500 4

S
B B B S C Type/ 802.2 Header
Field > IFG Preamble O TPID ITAG DA SA FCS
DA SA TAG TAG TAG Length and Payload
F

Overhead = 68 Bytes
 Ethernet OAM

Key Capabilities:
End to End Service
Performance Management
End to End Service
Fault Management
Point to Point
Link Fault Management

Source: Metro Ethernet Forum (www.metroethernetforum.org)

 Ethernet OAM in Action

E-LMI E-LMI

Carrier Ethernet
Ethernet Network Ethernet

Link Link
OAM OAM

Link Link Link

OAM OAM OAM
MIP MIP
Down Down
MEP MEP
Customer Domain
Up Up
MEP MEP
SP Domain

Acronyms: MEP - Maintenance End Point MIP - Maintenance Intermediate Point E-LMI - Ethernet Local Management Interface

• Y.1731: Fault Management (FM) and Performance Management (PM)

• 802.3ah - Link OAM: Link level monitoring and signaling.
• E-LMI: PE-CE communication of service status and attributes.
Ethernet and Transport OAM Comparison

SONET/SDH Ethernet Notes

Device Configuration Minimal Heavy SONET/SDH works

“out-of-the-box”.
Ethernet requires configuring
MEPs/Probes per service

How OAM Information In every frame Sent on dedicated Specific fames for Fault Management
Is Sent frames and Performance Management

OAM Frame 125us 1 ms to 1 minute Ethernet FM/PM Frame processing

Frequency requires CPU or specialized hardware

OAM Overhead ~4% ~5% 10ms/2500 MEPs/256B

messages/10Gbps interface =
512Mbit/s

CFM Connectivity Check Messages

Only
Adding Transport Encapsulation to Ethernet with G.709
• Available in new generation of Routers, Carrier Ethernet Switches and Packet
Transport equipment.
• Suitable to DWDM applications leveraging FEC, colored optics and dark fiber.
• Potentially PoS performance at price points closer to native Ethernet.

G.709 Enabled Ethernet Interface Architecture

Optical
Ethernet
Packet G.709 I/F
MAC
Processing Framer (Grey/
Layer
Colored)
Achieving 10GE LAN PHY Line Rate over G.709
Standard G.709

Overclocking

10GBASE-R Bit Rate Coded with 64B/66B: 10,3125 Gbit/s

G.709 (ODU2) Payload: 9,995 Gbit/s
Solution: Overclocked G.709 Frame (ODU2e)
IP/MPLS Services for Ethernet
• IP/MPLS provides the most comprehensive services to Ethernet.
• VPWS: Pseudowires for point to point services;
• VPLS: full mesh of Pseudowires to provide multipoint services;
• EVPN as the latest addition, leverages BGP control plane.

• Latest standards further enhance carrier grade features of IP/MPLS:

• TI-LFA: Topology Independent Loop-free Alternate, provides sub-50ms convergence;
• SR: Segment Routing, removes protocols for simpler deployment;
• RFC 3107: Hierarchical LSPs to scale MPLS domain to 100’s of thousands of nodes.
• BGP-PIC: Prefix Independent Convergence, predictable convergence times.
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
Needs for 40GE & 100GE optical modules
Reach Choice of optimized vertices will
vary by end user & segment of the
network

Cost Tradeoff Core

Triangle

Distribution

Density
Access
Seems impossible to optimize all
3 vertices simultaneously… Servers
& Clients
40/100G Client and Transport Reference

Transport Network (OTN/DWDM)

Regen
OA OA OA OA

TxP

Mux/Demux
ROADM

Client Optics Line Optics

• Within a building/campus (< 10km) • Across city/state/country (100s or 1000s km)
• Single I/F / Fiber (Grey optics) • Multiple channels / Fiber (DWDM)
• POS, OTN & Ethernet • OTN
• Cost is King • Performance is King
40G/100G DWDM Optics
• Expect 40G/100G Deployments to:
•Target 10G distances—1500 - 2000 Km
reach
•Plug-and-play over existing 10Gig networks
•Must be spectrally efficient- 50GHz Grid
•Power / density / cost / performance trade
off

• As Bit Rate increases the above

becomes more challenging
Simplify deployment of 100Gig into 10Gig Systems
• Reach & bandwidth
Choosing the Best Modulation Format
From 2.5Gb/s to 100Gb/s:
2.5G 10G 40G 100G > 100G
2.5G NRZ 10G NRZ 40G ODB 100G CP-QPSK --- ? ---
ASK ASK 40Gbs, ASK + PSK 25Gbaud, pol mux,
Variant: Coherent Rx
Chirp Most likely
Electronic equalizer
MLSE
40G DPSK Multilevel Tx
40Gb PSK
Polarization mux
Coherent Rx
40G A-DPSK
40Gb PSK
A=Adaptive
e.g.
28 Gbaud 16QAM
40G RZ-QPSK
20Gbaud, Q-PSK, RZ --- ? ---

40G CP-QPSK
10Gbaud, pol mux,
Coherent Rx

Only ONE modulation format is considered in the industry for 100G “lessons learned” from 40G
OIF activity to standardize the interface, components, …
 Flex Mod
Nyquist shaping and software configurable modulation

50G PM-BPSK

100G PM-QPSK

200G PM-16QAM

28 Gbaud/s
Nyquist shaped
Coherent Optical Transmission Benefits
High Chromatic Dispersion (CD) Robustness
• Can avoid Dispersion Compensation Units (DCUs)
• No need to have precise Fibre Characterization
• Simpler Network Design

High Polarization Mode Dispersion (PMD) Robustness

• High Bit Rate Wavelengths deployable on all Fibre types
• No need for “fancy” PMD Compensator devices
• No need to have precise Fibre Characterization

Low Optical Signal-to-Noise Ratio (OSNR) Needed

• More capacity at greater distances w/o OEO Regeneration
• Possibility to launch lower per-channel Power
• Higher tolerance to Channels Interferences
How to increase the port density at high speeds?
• Solution must address power and footprint requirements;
• CMOS photonics;
• Next-generation transceivers:
• CPAK;
• CFP2, CFP4;
• CXP2;
• QSFP28;
• xXP;
• Systems on a chip – integrate more functions in less ASICs.
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
High-Level Comparison - End-User View
Best Worst
Cost Scalability/ Efficiency Operat. E2E Compat./
Speed L3/L1 Simplicity OAM Interop. with
(IMIX) Transport
Ethernet
(LAN-PHY)

Ethernet
(WAN-PHY)

Ethernet
(w/ Ethernet OAM)

Eth. + G709
(Integrated)

POS
L3 Throughput Overview: Comparing 10Gbps Options Reference

L3 Data Rate for Different 10Gbps Transport Technologies

by packet sizes
10,000,000

9,000,000

8,000,000

7,000,000
L3 Data Rate (kbit/s)

6,000,000

5,000,000

4,000,000

3,000,000

2,000,000

1,000,000

0
40 64 128 256 512 1024 1518 9618
Packet Size (bytes)

PPP/HDLC Cisco HDLC Ethernet over ODU2 with GFP

Ethernet LAN-PHY Ethernet WAN-PHY Ethernet LAN-PHY 802.1Q
Ethernet WAN-PHY 802.1Q Ethernet LAN-PHY 802.1ad Ethernet LAN-PHY 802.1ah

Note: Ethernet interfaces don’t take into account EOAM frames which can be considered additional overhead.
 Reference
L3 Throughput Overview (IMIX):
Comparing Types of 10Gbps Interconnections
L3 Data Rate for Different 10Gbps Transport Technologies
for IMIX Traffic
10,000.00 100.0%

9,500.00 95.0%

L3 Data Rate - Mbit/s 9,000.00 90.0%

Efficiency %
8,500.00 85.0%

8,000.00 80.0%

7,500.00 75.0%

7,000.00 70.0%
PoS PoS (Cisco LAN-PHY LAN-PHY with WAN-PHY WAN-PHY
(PPP/HDLC) HDLC) 802.1Q with 802.1Q

Total L3 Throughput (Mbit/s) Efficiency - L3 Data Rate vs Link Payload Capacity

Note: Ethernet interfaces don’t take into account EOAM frames which can be considered additional overhead.
Agenda

• Introduction
• Traditional High Speed WAN Technologies
• Ethernet in the WAN
• High-Speed Optics
• Evaluation of Key Attributes for WAN
• Summary
Summary
• Business requirements are driving higher speeds in the WAN.
• Higher speeds can lead to higher cost and more complexity, pushing for
technology innovations and new service paradigms.
• There are many options to migrate to higher speeds in the WAN, each one has
it’s pros and cons. Goal is to understand and minimize the impact of the
tradeoffs.
• Optical layer has evolved to support higher speeds (40Gbps and above) by
adopting advanced modulations techniques or multiple lanes.
• There is a much more clear separation between client side optics and line side
optics at speeds of 40Gbps and above.
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
Thank you
WAN Branch Speed Mix

Source: Aryaka – State of Global Enterprise WAN Report - 2014

WAN Branch Speed Mix

Source: Aryaka – State of Global Enterprise WAN Report - 2014

 Acronyms
APC Automatic Power Control
BFD Bidirectional Forwarding Detection Protocol (IETF RFC 5880)
CAPEX Capital Expenditure
CRC Cyclic Redundancy Check
(D)WDM (Dense )Wave Division Multiplexing
E-LMI Ethernet Link Management Interface (Metro Ethernet Forum standard)
FCS Frame Check Sequence
FM Fault Management
GACH Generic Associated Channel (ref. MPLS-TP)
GE Gigabit Ethernet
GFP Generic Framing Procedure
GMP Generic Mapping Procedure
GMPLS Generalized Multiprotocol Label Switching
HDLC High-level Data Link Control Protocol
IA Implementation Agreement (related to MEF)
IGP Interior Gateway Protocol
IMIX Internet Mix (Mix of different IP packet sizes in typical Internet traffic)
ITU International Telecommunications Union
LAN Local Area Network
LSP Label Switched Path
 Acronyms
MAC Media Access Control
MAN Metropolitan Area Network
MEF Metro Ethernet Forum (www.metroethernetforum.org)
MP Multipoint
MPLS Multiprotocol Label Switching
MPLS-TP Multiprotocol Label Switching – Transport Profile
NNI Network-to-Network Interface
OAM Operations, Administration and Maintenance
OPEX Operational Expeditures
OTN Optical Transport Network
P2MP Point-to-Multipoint
P2P Point-to-point
PHY Physical Layer
PM Performance Management
PMD Physical Medium Dependant (related to IEEE 802.3 Ethernet Interfaces)
POS Packet over SONET/SDH
PPP Point-to-point Protocol
PW Pseudowire
SOAM Service OAM
 Acronyms
SONET/SDH Synchronous Optical Network Transport/Synchronous Digital Hierarchy
STP Spanning Tree Protocol
UNI User-to-Network Interface
VC Virtual Container
WAN Wide Area Network