Layer 2/3/QoS

Test Methodology

www.spirentcampus.com
Topic Overview
„ Types of Tests
„ What is Performance Testing?
„ BMWG – Benchmarking Methodology Working Group
„ Core RFCs (1242, 2544, 2285, 2889)
„ Evolution of Switching Technology
„ Ethernet, VLAN, IPv4, QoS Test Methodologies
„ Test Objectives
„ Test Parameters
„ Test Procedures
„ Reporting Formats
2
Layer 2/3/QoS Test Methodology
Types of Tests
„ In exercising and validating all the characteristics of a DUT,
there are four different types of tests to run:
„ A Conformance test consists of verifying the device’s
behavior in correspondence with the standards, conventions,
rules.
„ A Functional test consists of verifying that the device does
everything it is supposed to do (i.e., protocol support, filters,
management, etc.).
„ A Performance test consists of “stress testing” the device to
see how it behaves under loaded conditions.
„ A Passive test consists of Protocol analysis. This can be
accomplished either intrusively or non-intrusively.

3
Layer 2/3/QoS Test Methodology
Conformance Testing

„ Validating adherence to a standard

„ Tests all facets of the RFCs
„ Concepts developed by Telcordia – Bell Standard
„ Hundreds of individual tests
„ Identifies the sections of the RFC that are being tested
„ Assures correct handling of error conditions
„ Should reveal how the “SHALLs” and “MAYs” are
implemented (i.e., optional elements)
„ Subset of these tests can be used as manufacturing test
(e.g., does a specific board meet a physical interface
standard?)
4
Layer 2/3/QoS Test Methodology
Functional Testing

„ Validates “system” components

„ Internal management functions
„ Control plan and data plane functions
„ Proprietary functions between systems (e.g. encapsulation,
trunking protocols)
„ Testing of the correct implementation and operation of a
communications protocol
„ Typically less rigorous than a conformance test
„ Verifies interaction between multiple functions
„ Negative Testing (e.g., introduce impairments to a stable
device or network)
5
Layer 2/3/QoS Test Methodology
Functional Testing – Negative Testing

Introduce impairments to a stable device or network:

„ Route or link flapping
„ Invalid routes
„ Malformed packets
„ Invalid policies
„ Lost keep-alives

6
Layer 2/3/QoS Test Methodology
Performance Testing
„ What is performance testing?
‹ Setting an expectation of the device’ functionality
„ It could be said, “there are 3 levels of performance testing.”
‹ Performance testing
‹ Load testing
‹ Stress testing
„ Keep it realistic.
‹ A broad distribution of source and destination addresses.
‹ A mix of packet sizes and types.
‹ The bandwidth characteristics of realistic traffic.
‹ Different ratios of unicast and multicast traffic.
‹ Many different streams and conditions that force prioritization (for QoS)
7
Layer 2/3/QoS Test Methodology
Why Test Maximum Device
Configurations?
„ “Until you fully load a device, you won’t find the
problems.”

8
Layer 2/3/QoS Test Methodology
Who Needs Performance Testing?
„ Test setups and methodologies were derived from the
following RFCs:
‹ RFC 1242 “Benchmarking Terminology for Network Interconnection Devices”
‹ RFC 2544 “Benchmarking Methodology for Network Interconnection Devices”
‹ RFC 2285 “Benchmarking Terminology for LAN Switching Devices”
‹ RFC 2889 “Benchmarking Methodology for LAN Switching Devices”
Performance Tester

Traffic Traffic
Generation: Analysis:
X Frames Y Frames
Device Under Test (DUT) Received
Transmitted

9
Layer 2/3/QoS Test Methodology
Benchmarking Methodology Working
Group (BMWG)
„ The BMWG makes recommendations concerning the measurement of
the performance characteristics of various internetworking technologies.
‹ Describes the class of equipment, system, or service being addressed
‹ Discusses the performance characteristics that are pertinent
‹ Identifies a set of metrics that aid in the description of those characteristics
‹ Specifies the methodologies required to collect said metrics
‹ Presents the requirements for the common, unambiguous reporting of
benchmarking results
„ The scope of the BMWG is limited to technology characterization using
simulated stimuli in a laboratory environment.
„ The benchmarks shall strive to be vendor independent or otherwise have
universal applicability to a given technology class.
http://www.ietf.org/html.charters/bmwg-charter.html

10
Layer 2/3/QoS Test Methodology
Core RFC Parameters

„ Test setups and methodologies were derived from the

following “Core” RFCs:
‹ RFC 1242 “Benchmarking Terminology for Network Interconnection
Devices”
‹ RFC 2544 “Benchmarking Methodology for Network Interconnection
Devices”
‹ RFC 2285 “Benchmarking Terminology for LAN Switching Devices”
‹ RFC 2889 “Benchmarking Methodology for LAN Switching Devices”

11
Layer 2/3/QoS Test Methodology
Core RFC Parameters (Cont.)

„ Test Duration
„ Number of Trials
„ Addresses Per Port
„ Address format
„ Frame Sizes
„ Frame Format
„ Burst Size
„ Load Percentage

12
Layer 2/3/QoS Test Methodology
Test Trial Description

„ Following is a list of the steps that are followed in performing

a test trail:
‹ Reset Ports
‹ Test setup
‹ Sending Learning Packets
‹ Receive Learning Results
‹ Transmitting
‹ Receiving Test Results
‹ Repeat based on user selectable parameters

13
Layer 2/3/QoS Test Methodology
Layer 2 Learning

„ MAC Address table updating occurs at the beginning of a

test (i.e., during the Learning Phase).
„ During bi-directional traffic the table update will occur in both
directions.
Port 1 Port 2

Test Port #1 DUT= Layer 2 Bridge Test Port #2

(DA) MAC=000000000002 Bridge MAC Address Table (SA) MAC=000000000002

Port 2=MAC 000000000002

1. Port # 2 Sends a Learning Packet to Bridge

2. Bridge’s MAC Address Table is Updated

3. Port #1 Sends Traffic to Port #2

14
Layer 2/3/QoS Test Methodology
Settings – L2 Learning

„ Layer 2 Learning teaches the DUT the source MAC address

of streams configured on the test port.

15
Layer 2/3/QoS Test Methodology
Layer 3 Learning
„ The exchange below occurs at the beginning of a test.
„ This exchange describes traffic in only one direction.
„ For bi-directional traffic the exchange will occur in both directions.
Port 1 Port 2

Test Port #1 DUT=Layer 3 Router Test Port #2

Protocol=IP
MAC Dest=080002068922
MAC=000000000001
Card IP=192.168.99.2
Router’s IP=192.168.99.1
Protocol=IP
MAC Port1=080002068922
IP Port1=192.168.99.1
MAC Port2=080002898856
IP Port2=192.168.88.1
Protocol=IP
MAC Dest=080002898856
MAC=000000000002
Card IP=192.168.88.2
Router’s IP=192.168.88.1

1. Port #1 Sends a Packet to Port #2 2. Causes Router to ARP Port #2

3. Port #2 Sends an ARP Reply

4. Router’s ARP Table is Updated

5. Port #1 Sends test Packets to Port #2
16
Layer 2/3/QoS Test Methodology
Settings – ARP / ND

„ The ARP/ND functionality resolves the MAC address of the

gateway/destination configured in stream blocks or emulated
devices.

17
Layer 2/3/QoS Test Methodology
Data vs. Control Planes
„ Data and Control planes are closely tied together.
„ Data plane (e.g., Forwarding Information Base or FIB)
‹ Responsible for packet forwarding, QoS enforcement.
„ Control plane (e.g., Routing Information Base or RIB)
‹ Processes routing protocol messages, builds the forwarding table,
and advertises routing information to other routers.

Control Plane Data Plane

Higher Layers Higher Layers

IP + ARP Layer 3

Datalink Layer 2

Physical Layer 1

18
Layer 2/3/QoS Test Methodology
Performance vs. Reliability

„ Performance is important, but reliability is also as important.

„ Reliability issues include:
‹ That interface card failures do not affect the operation of the router.
‹ That switching fabric card failures predictably diminish a router’s
performance.
‹ That a router does not fail during times of route instability.
‹ That the device recovers from power failures quickly and does not
require manual intervention.
‹ Also includes testing fail-over mechanisms.

19
Layer 2/3/QoS Test Methodology
Per Port vs. Per Stream Testing

20
Layer 2/3/QoS Test Methodology
Per Port Testing

„ “Packet blasting” – per port traffic loading and measurement.

„ Per Port metrics include: throughput, latency, and packet
loss.

A legacy product is depicted intentionally to

indicate that this type of testing is also legacy

DUT

21
Layer 2/3/QoS Test Methodology
Per Stream Testing

Single 10/100 Mbps Stream-based test port:

„ Emulates 100s of end-stations
„ Tracks QoS for each stream
„ Creates deterministic repeatable performance test
VOIP

„ Each frame in each stream stamped with a unique

signature for advanced tracking capabilities
Test
Port

Laptop computer

Video

server
The “wire”
22
Layer 2/3/QoS Test Methodology
Per-Stream Metrics: Sequence
Tracking
Tracking a sequence of packets interspersed with other traffic

12 14 13 15

12 13 14 15 12 14 13 15

DUT

23
Layer 2/3/QoS Test Methodology
Latency Test using Timestamps
„ Every Frame is individually time stamped.
„ Tester can compare transmit and received timestamps.
„ More advance latency measurements can be discerned including
min/max/avg., distributions, and variations.

Performance Tester
Signature
Frames
DUT

24
Layer 2/3/QoS Test Methodology
Per-Stream Metrics: Latency Variation

„ Latency variation affects voice and video quality

Layer 3 multi-media streams

12 13 14 15

Other layer 3
streams

12 13 14 15

25
Layer 2/3/QoS Test Methodology
How Latency is Measured in the
Different Modes
„ For just Latency, it can either be start or end of frame (Rx
minus Tx time stamp).
‹ This is user configurable on both the Tx and Rx side.

„ There is a difference in the way hardware measures Jitter

and Inter-arrival Time.
„ For Jitter, it is the difference in delay of two frames
‹ Requires 4 parameters to calculate – transmit and arrival times for
two frames (the delta of the deltas)

„ Inter-arrival time, uses just the delta time between frames

‹ Fails when packet loss occurs because the two frames are most
likely farther apart

26
Layer 2/3/QoS Test Methodology
 Calculating Jitter versus Inter-arrival
Time
„ Jitter is a calculation unique to Spirent TestCenter
„ It requires a specific hardware design whereas you can hold two values in
memory and then perform a calculation on them in real time, separately for
potentially 64,000 individual streams, and at up to 15 billion frames per
second!
„ Jitter = the delay of the previous frame minus the delay of the current one.
‹ Only measured for frames in sequence.
‹ Does not matter that the t3 frame was lost below.
„ Inter-arrival Time = the time between receiving the last bit of the previous
frame and the first bit of the current frame.
‹ So in the example sequence below, you would have a greater inter-arrival time
between t2 and t4 then you did between t1 and t2.
‹ It is also affected by the rate which Jitter is not.
t1 t2 t3 t4 t5 t6 t7 t8 t9 t10
x x x x x x x
27
Layer 2/3/QoS Test Methodology
Bottom up Testing

„ Test the foundation first.

„ Work your way up the OSI model.

28
Layer 2/3/QoS Test Methodology
Systematic Performance Benchmarking

Logical steps:
„ Benchmark using a single stream.
„ Load with uni-directional traffic using multiple port pairs.
„ Make it bi-directional, again using multiple port pairs.
„ Try various partially meshed traffic patterns.
„ Load with a fully meshed traffic pattern.

29
Layer 2/3/QoS Test Methodology
How is Throughput Measured?
„ Percentage of theoretical maximum is based on one of the following:
‹ Frames per second (Ethernet Layer 2) and Packets per second (IP Layer 3)
are interchangeable (i.e., 1:1 ratio)
‹ Bits per second
• Total bits = Min IFG + Preamble + Frame Length
• Frame bits = Frame Length (including L2 encapsulation)
• Data bits = Frame Length – L2 encapsulation (Datagram length)

Percentage of theoretical maximum

Frames per second

Total bits

IFG or IBG depending on Generator setting

Frames bits
30
Layer 2/3/QoS Test Methodology
Calculating Maximum Frame Rates
for Ethernet
Speed 10,000,000 bps
Max Rate 14,880 fps
Frame + Preamble + Bit Time
Size IFG
512 bits 64 bits 96 bits

672 bits

Size * 10MB 100MB Gigabit 10 Gigabit

(bytes) (fps) (fps) (fps) (fps)
64 14,880 148,809 1,488,095 14,880,952
128 8,445 84,459 844,594 8,445,945
256 4,528 45,289 452,898 4,528,985
512 2,349 23,496 234,962 2,349,624
1024 1,197 11,973 119,731 1,197,318
1280 961 9,615 96,153 961,538
1518 812 8,127 81,274 812,743

*7 recommended frames sizes to test from RFC 2544

31
Layer 2/3/QoS Test Methodology
Viewing Test Results – Port Counters

„ Port-based (aggregate) traffic

„ Includes all test streams, control plane and superfluous
traffic
„ Includes multiple tabs for viewing different classes of results

32
Layer 2/3/QoS Test Methodology
Viewing Test Results – Stream Statistics

„ Individual Stream results can be viewed in real-time or

aggregated at the end of the test.
„ The individual stream statistics are broken out separately;
they can be sorted by stream ID or filtered by some user-
slectable criteria (QoS bit, VLAN tag, IP address, etc.)

33
Layer 2/3/QoS Test Methodology
Viewing Test Results – Control Plane

„ The Protocols available depend on the selections made in

the Test.

34
Layer 2/3/QoS Test Methodology
Determining the Test Methodology

„ What do you want to test?

„ Why do you want to test it?
„ How are you going to test it?
„ What are the Parameters and Variables?
„ What about the Units?
„ How will you display the Results?
„ Develop a Plan and Map it out.
„ Execute the Plan.

35
Layer 2/3/QoS Test Methodology
What do you want to test?

„ At which layer(s) are at which point in the network does your

device perform?
Source Destination
Are you ready to accept a file? OK!!

Are the formats the same? OK!!

Setup Logical Connection OK!!

Have you received everything intact? OK!!

What is the address?

Setup Link Connection, Check for Bit Errors

Transport Bits

Communication Path Communication

Links

36
Layer 2/3/QoS Test Methodology
Number and Types of Interfaces

37
Layer 2/3/QoS Test Methodology
Test Setup: Single Chassis, Multiple
DUTs
Test Chassis

Traffic Traffic
Generation Analysis

DUT DUT
Different Media
Technology

SUT

38
Layer 2/3/QoS Test Methodology
System/Network Topology

„ How does everything tie together?

Edge Edge

Distribution Distribution

Core

39
Layer 2/3/QoS Test Methodology
Protocol Interactions

„ How do the protocols work together?

RTP/RTCP
TCP/UDP
IP
L2 Protocols

PPP/FR/POS
MPLS
RSVP/Diffserv
Ethernet 802.1p/Q

40
Layer 2/3/QoS Test Methodology
Why do you want to test it?

Depends upon your Organization and Job Function

„ Network Equipment Manufacturer (NEM)
‹ Development
‹ Engineering
‹ Quality Assurance (QA)
‹ Production
‹ Support

„ Enterprise
„ Service Provider

41
Layer 2/3/QoS Test Methodology
How are you going to test it?
Apply these tests …

Packet/Frame Tests Stream Tests

Aggregate Port Statistics Individual Stream Statistics
RFC 1242/2544/2285/2889 tests RFC ???? tests
… to these
Per port Throughput, Latency, Per stream Sequence Tracking,
types of
Packet Loss, Back-to-Back Latency Distribution and Variation
networks

Layer 2 Raw Layer 2 Forwarding Layer 2 Optimization such as:

networks based on destination MAC • MAC and Layer 3 Protocol Filters
• Layer 2 QoS control mechanisms

Layer 3 Raw Layer 3 Forwarding Layer 3 Optimization such as:

networks based on destination IP • IP and Layer 4 Protocol Filters
• Layer 3 QoS control mechanisms

Layer 4 Raw Layer 3/4 Forwarding Layer 3/4 Optimization such as:
networks based on a combination of • Stateful vs. Stateless inspection
s&d IP and s&d TCP/UDP ports • L3/4 QoS control mechanisms
42
Layer 2/3/QoS Test Methodology
What are the Parameters and
Variables?
„ Multi-Layer performance testing is significantly more
complex.
„ Without certain key information about the test, the results are
meaningless.
‹ What was the speed and duplex of the ports?
‹ What packet size(s) and rate(s) were used?
‹ What was the packet-loss tolerance?
‹ How many and what type of streams were used?

„ How do you determine these parameters?

„ You can start with the RFCs.

43
Layer 2/3/QoS Test Methodology
Request For Comments (RFCs)
„ Started in 1969
„ Technical notes about the Internet.
„ Defines standards and the process of creating them.
http://www.isoc.org/
http://www.iab.org/
http://www.ietf.org/
http://www.ietf.org/iesg.html
ftp://ftp.isi.edu/in-notes/rfc2026.txt
ftp://ftp.ietf.org/ietf/1id-guidelines.txt
http://www.postel.org/postel.html
44
Layer 2/3/QoS Test Methodology
RFC Compliancy

„ "MUST"
MUST is an absolute requirement of the specification.
„ "SHOULD"
SHOULD means that there may be valid reasons to ignore
this item.
„ "MAY"
MAY means that this item is truly optional.
„ An implementation that satisfies all the MUST and all the
SHOULD requirements for its protocols is said to be
"unconditionally compliant"

45
Layer 2/3/QoS Test Methodology
What about the Units?
„ Depending on the type of test it will probably be one of these:
‹ Data sent
‹ Percentage through
‹ Offered vs. loss.
‹ Time
• between events.
• nanoseconds, microseconds, milliseconds, or seconds.
‹ Ratio of input and output data rates.
‹ Description of behavior under various conditions.
‹ Address capacity
‹ Address Rate

46
Layer 2/3/QoS Test Methodology
How will you display the Results?

„ Group the results from multiple associated flows into a single

aggregate total.
„ This makes it easy to track associated traffic patterns and/or
parameters.
„ What do you expect the results to be?

47
Layer 2/3/QoS Test Methodology
Develop a Plan and Map it out
„ Test Methodology Example (RFC 2544)
‹ Objective
‹ Setup Parameters
‹ Procedure
‹ Measurements
‹ Reporting Format

48
Layer 2/3/QoS Test Methodology
Test Methodology Considerations

„ The following must be considered when formulating the test

plan:
‹ Test Lab Configuration
‹ Test Chassis and Device Port Assignments
‹ Test Numbering Scheme
‹ Device Configuration
‹ Traffic Pattern
‹ Test Application Configuration
‹ Test Execution Instructions
‹ Test Results Expectation/Display

49
Layer 2/3/QoS Test Methodology
Execute the Plan

Steps to Set up and Run a Test:

1. Configure the DUT/SUT/NUT.
2. Connect your test chassis.
3. Setup the communications between the test application and
the test chassis.
4. Configure the port’s “base” parameters.
5. Configure the traffic pattern and parameters.
6. Setup the general test configuration parameters and
preferences.
7. Run the test.
8. Analyze and save the results.
50
Layer 2/3/QoS Test Methodology
Evolution of Switching Technology

„ Following, we will evaluate functionality specified

by the associated Benchmarking RFC
QoS
„ Spirent TestCenter can be used to test in
all of these test environments
L4-7
Switch
Bridge L2 L3
VLAN
Router Switch Switch
VPNs
Increased intelligence

Multicast
IP
80s Early to Mid 90s Mid 90s Mid to Late 90s Y2K and beyond
51
Layer 2/3/QoS Test Methodology
Bridge/Router Testing

„ RFCs that describe the Terminology/Methodology for testing

a CPU-based Bridge or Router are:
‹ RFC 1242: Benchmarking Terminology for Network Interconnect
Devices
‹ RFC 2544: Benchmarking Methodology for Network Interconnect
Devices
OSPF

BGP
Bridge L2 L3
Router VLAN
Switch Switch

MPLS

Multicast
IP
52
Layer 2/3/QoS Test Methodology
RFC 1242 Overview

„ Authored by Scott Bradner of Harvard University in July

1991.
„ Defines terms used describing performance benchmarking
tests and the results of such tests.
„ Some of the terms defined by RFC 1242 are:
– Back-to-back – MTU-mismatch behavior
– Constant Load – Overhead behavior
– Data link frame size – Overloaded behavior
– Fragment – Policy based filtering
– Frame Loss Rate – Restart behavior
– Inter Frame Gap – Router
– Latency – Single frame behavior
– Link Speed Mismatch – Throughput

53
Layer 2/3/QoS Test Methodology
RFC 2544 Overview
„ RFC 2544 "Benchmarking Methodology for Network Interconnect
Devices" defines a specific set of tests that vendors can use to measure
and report the performance characteristics of network devices.
„ The results of these tests will provide the user comparable data from
different vendors with which to evaluate these devices.
„ RFC 1242 should be consulted before attempting to make use of this
document.
„ Provides a methodology for benchmarking bridges or routers:
‹ Packet Loss
‹ Throughput
‹ Latency
‹ Back-to-Back
‹ System recovery
‹ Reset

54
Layer 2/3/QoS Test Methodology
RFC 2544 Summary
„ Suggested frame sizes for Ethernet testing:
‹ 64, 128, 256, 512, 1024, 1280, 1518
‹ At least five frame sizes SHOULD be tested for each test condition
„ The Throughput test should be at least 60 seconds per iteration
„ The Latency test should be at least 120 seconds per iteration and must be run at
least 20 times with average taken. Also, it should run at the determined
Throughput rate.
„ Frame Loss rate resolution must not be greater then 10%
„ The Back-to-Back test trial length must be at least 2 seconds and should be
repeated at least 50 times with the average taken
„ All tests should be run with and without filters enabled on the DUT
‹ Either Layer 2 MAC or Layer 3 Protocol addresses depending on the DUT
„ Traffic should be sent Bi-directionally
„ An application level datagram echo request is used for the test data frame in the
protocols that support such a function
‹ For TCP/IP a UDP Echo Request is used
55
Layer 2/3/QoS Test Methodology
Throughput Test
„ If X = Y increase the frame rate
„ If X > Y decrease the frame rate
„ A Binary Search Algorithm is used to determine the highest frame rate at which
X=Y
Performance Tester

Traffic Traffic
Generation: Analysis:
X Frames Y Frames
Transmitted DUT Received

56
Layer 2/3/QoS Test Methodology
Throughput Test Setup

„ Duration is recommended to be > 60 sec; + Number of Trials

57
Layer 2/3/QoS Test Methodology
Throughput Test Setup (Cont)

„ Frame Sizes: Start/Step/Stop or Custom

58
Layer 2/3/QoS Test Methodology
Throughput Test Setup (Cont)

„ Binary Mode Rates: Initial Rate, Upper and Lower limit

„ Resolution and Acceptable Loss
„ Try to fine-tune the algorithm based on your needs

59
Layer 2/3/QoS Test Methodology
Throughput Test Setup (Cont)

„ Learning Mode and Retries

60
Layer 2/3/QoS Test Methodology
Throughput Test Results

61
Layer 2/3/QoS Test Methodology
Latency Test
„ Frame Rate <= Throughput Rate
„ Tagged Frame is inserted into the middle of the transmitted packet stream
„ The Latency of the Tagged Frame is measured and recorded as the FIFO
„ The bit time of the Tagged Frame is subtracted from the FIFO to determine LIFO

Performance Tester

Tagged
Frame

DUT

62
Layer 2/3/QoS Test Methodology
Latency Test per RFC 2544

Input packet

Rx Timestamp

Output packet
(First bit of receive
Tx Timestamp
packet)
(First bit of
transmit packet)
DUT

First-in/First-out (FIFO) Time is: (Rx Timestamp) - (Tx Timestamp)

This is Reported as “Cut-Through Latency”
Last-in/First-out (LIFO) Time is: (FIFO Time) - (Frame bit-time)
This is Reported as “Store-and-Forward Latency” 63
Layer 2/3/QoS Test Methodology
Switching Modes Affect on Latency

Frame In ...... Frame Out

128 byte frame

DA SA 64 bytes FCS DA SA ...

Cut-Through time

Fragment Free time

Store-and-Forward
Store time (Frame TAX)

First In First Out Last In First Out

Device
Hold
time
64
Layer 2/3/QoS Test Methodology
Latency Test Setup

„ Duration is recommended to be > 120 sec; + Number of Trials

„ Also, select the Latency Type

65
Layer 2/3/QoS Test Methodology
Latency Timestamp Reference
Location
„ It is possible to set the Latency Timestamp Reference
Location different of each side.
„ However, for most test cases, both sides should be set to
Start of Frame which is the default.

66
Layer 2/3/QoS Test Methodology
Packet Loss Test
„ Frame Loss Rate = (X-Y) / X * 100
„ Measured a various user selectable Frame Rates

Performance Tester

Traffic Traffic
Generation: Analysis:
X Frames Y Frames
Transmitted DUT Received

67
Layer 2/3/QoS Test Methodology
Back-to-Back Test
„ Determine the largest Burst Size the DUT can handle with “zero” packet loss
„ If no frames are dropped, increase the Burst Size
„ Spirent allows user selectable rates, but the RFC 2544 states minimum gap

Performance Tester

Minimum
Interframe Burst Size,
Gap Duration

DUT

68
Layer 2/3/QoS Test Methodology
System Recovery
„ To determine how fast a DUT recovers from an overload condition.
„ The Systems Recovery test is performed as follows:
‹ Find the throughput for each frame size.
‹ Stream frames at 110% of the recorded throughput rate for at least
60 seconds.
‹ Reduce the frame rate to 50% of the above rate and record the time
of the last frame lost (Timestamp B).
‹ Subtract Timestamp B from Timestamp A.
‹ Repeat a number of times and average the recorded values
reported.
FRAME SIZE RATE RECOVERY TIME
64 90% 2 seconds
1518 110% 1 second

69
Layer 2/3/QoS Test Methodology
Reset

„ Determines how quick a DUT recovers a reset.

„ The Reset test is performed as follows:
‹ Find the throughput for the DUT’s minimum Ethernet frame size.
‹ While sending a continuous stream of frames, reset the DUT.
‹ Record the time that the last frame before the reset was received
(Timestamp A), and subtract it from the time that the first frame after
the reset is received (Timestamp B).
‹ A power interruption reset test is performed as above except that the
power to the DUT should be interrupted for 10 seconds in place of
causing a reset.
‹ This test should only be run using frames addressed to networks
directly connected to the DUT.

70
Layer 2/3/QoS Test Methodology
Layer 2 Switch Testing
„ RFCs that describe the Terminology/Methodology for testing
a Layer 2 Switch are:
‹ RFC 1242, RFC 2544 Generally
‹ RFC 2285: Benchmarking Terminology for LAN Switching Devices
‹ RFC 2889: Benchmarking Methodology for LAN Switching Devices

QoS

L4-7
Bridge L2 Switch
VLAN L3
Router Switch Switch

VPNs

Multicast
IP

71
Layer 2/3/QoS Test Methodology
RFC 2285 Overview

„ Provides terminology for the benchmarking of local area

network (LAN) switching devices.
„ Extends terminology defined in RFCs 1242 and 2544 to
switching devices.
„ Some of the terms defined by RFC 2285 are:
¾ Device under test (DUT) Forwarding rates
¾ System under test (SUT) Congestion control
¾ Unidirectional traffic Backpressure
¾ Bi-directional traffic Forward pressure
¾ Non-meshed traffic Head of line blocking
¾ Partially meshed traffic Address handling
¾ Fully meshed traffic Errored frame filtering
¾ Bursts Broadcasts
¾ Loads

72
Layer 2/3/QoS Test Methodology
DUT, SUT, or NUT
Device Under Test
DUT

System Under Test

SUT

Network Under Test

NUT

73
Layer 2/3/QoS Test Methodology
Traffic Distribution: Non-meshed

DUT/SUT

Port 1 Port 4

Port 2 Port 5

Port 3 Port 6

74
Layer 2/3/QoS Test Methodology
Traffic Distribution: Partially-meshed

DUT/SUT

Port 1 Port 4

Port 2 Port 5

Port 3 Port 6

75
Layer 2/3/QoS Test Methodology
Traffic Distribution: Fully-meshed

DUT/SUT

Port 1 Port 4

Port 2 Port 5

Port 3 Port 6

76
Layer 2/3/QoS Test Methodology
Fully Meshed Frame Distribution

Source Port Destination Ports (in order of transmission)

Port #1 2 3 4 5 6 2…
Port #2 3 4 5 6 1 3…
Port #3 4 5 6 1 2 4…
Port #4 5 6 1 2 3 5…
Port #5 6 1 2 3 4 6…
Port #6 1 2 3 4 5 1…

77
Layer 2/3/QoS Test Methodology
Burst Size
„ How many frames will be sent with minimum interframe gap?
„ There are two ways to specify a range of bursts:
‹ Start/Step/Stop method (RFC 2889 style)
‹ Custom method (RFC 2554 style)

This or That

RFC 2889 RFC 2544

78
Layer 2/3/QoS Test Methodology
Calculating the Interburst Gap (IBG)
„ IBG is defined in RFC 2285 as the interval between two bursts.
„ AS per RFC 2889, the IBG (in seconds) can be calculated as follows:

[(100/LOAD - 1) * BURST * (IFG + 64 + 8*LENGTH)] + IFG

IBG = ----------------------------------------------------------------------------------
SPEED

...........

IFG IBG

Test Trial Duration

79
Layer 2/3/QoS Test Methodology
Number of Bursts for the Trial Duration
„ AS per RFC 2889, the Number of Bursts for the Trial Duration can be calculated
as follows:
„ First, calculate an intermediate number of the Burst duration:
IFG*(BURST-1) + BURST*(64 + 8*LENGTH)
TXTIME = ----------------------------------------------------------- = Single Burst Time
SPEED

„ Then, calculate the number of Bursts for the Trial Duration (rounded up):
DURATION
# OF BURSTS = ---------------------
(TXTIME + IBG)

„ The number of bursts for the trial duration is rounded up to the nearest integer
number.
80
Layer 2/3/QoS Test Methodology
Generating Offered Load
„ The traffic generator is configured with the Intended Load (Iload) and
measures the Offered Load (Oload).
„ If the DUT/SUT applies congestion control (back-pressure), then these
two loads will not be the same value.
„ The question arises, how to generate the Offered Load?
‹ Frame-based tests hold the number of frames constant. The trial
duration will vary based upon congestion control.
‹ Time-based tests hold the trial duration constant, while allowing the
number of transmitted frames to vary.

Intended Load

Offered Load

81
Layer 2/3/QoS Test Methodology
Hypothetical Forwarding Rates Graph

„ MFR - Maximum Forwarding Rate

„ MOL - Maximum Offered Load
„ FRMOL - Forwarding Rate at Maximum Offered Load
MFR

FRMOL

Device
Forwarding
Rates

MOL
Offered Loads
82
Layer 2/3/QoS Test Methodology
Addresses Per Port

„ Allows you to simulate multiple Hosts from a single Test port

83
Layer 2/3/QoS Test Methodology
RFC 2889 Overview
„ Provides the methodology for the benchmarking of local area network
(LAN) switching devices.
„ Extends the methodology already defined for benchmarking network
interconnecting devices in RFC 2544 to switching devices.
„ Deals primarily with devices which switch frames at the Media Access
Control (MAC) layer.
„ Provides a methodology for benchmarking
switching devices:
‹ Forwarding performance
‹ Congestion control
‹ Broadcast Frame Throughput /Latency
‹ Address handling
‹ Error filtering
„ The following RFCs SHOULD be consulted before attempting to make
use of this document: RFC 1242, RFC 2285, and RFC 2544.
84
Layer 2/3/QoS Test Methodology
RFC 2889 Benchmarking Tests
„ Fully meshed throughput, frame loss and forwarding rates
„ Partially meshed one-to-many/many-to-one
„ Partially meshed multiple devices
„ Partially meshed unidirectional traffic
„ Congestion Control
„ Forward Pressure and Maximum Forwarding Rate
„ Address Caching Capacity
„ Address Learning Rate
„ Erred frames filtering
„ Broadcast frame Forwarding and Latency
85
Layer 2/3/QoS Test Methodology
Forwarding Test
„ Used to measure a devices throughput and/or packet loss

86
Layer 2/3/QoS Test Methodology
Congestion Control Methodology
„ Frames destined to uncongested ports in a switch device should not be
dropped due to other ports being congested, even if the source is
sending to both the congested and uncongested ports.
„ This test simulates a condition of congestion at one port while a lighter
load is presented to another uncongested port.
„ The diagram below depicts the flow of traffic between the switch ports:

50% Load Tester

Tester DUT DUT Uncongested
Transmitter 1 Port 1 Port 3 Receiver

50% Load

Tester
Tester DUT DUT 150%
Congested
Transmitter 2 Port 2 Port 4 Load
100% Load Receiver
87
Layer 2/3/QoS Test Methodology
Congestion Control Test
„ Used to determine how a DUT handles congestion
„ Also used to test 802.3x “pause” flow control (backpressure)

88
Layer 2/3/QoS Test Methodology
Address Handling Test Methodology
Determining the Max Number of Addresses
„ In this part of the test, the objective is to determine how many addresses the DUT can
handle per port. In order to do this, the application must be able to determine when the
DUT address table overflows. To determine this, the application uses a three port
configurations as described below:
„ Learn Port (01): This port (single entry, left side of graphic) is connected to the DUT and is
the port from which the application sends frames with varying source addresses and a fixed
destination address corresponding to test port 2. By sending frames with varying source
addresses, the DUT will learn these new addresses for the connected port.
„ Test Port (02): This port (single entry, right side of graphic) is connected to the DUT and
acts as the receiving port for the address learning frames. The second function of this test
port is to send "control" frames back to the addresses corresponding to test port 1.
„ Monitor (03): This port (single entry, centered below graphic) is also connected to a port on
the DUT and acts as a monitoring port (hence the satellite dish) and listens for frames
being flooded, indicating that the DUT address table is full.

89
Layer 2/3/QoS Test Methodology
Error Filtering Test

„ To determine the behavior of the DUT under error or

abnormal frame conditions.
„ The results of the test indicate if the DUT/SUT filters the
errors, or simply propagates the erred frames along to the
destination.

90
Layer 2/3/QoS Test Methodology
Broadcast Forwarding and Latency

„ Used to determine the throughput and latency of the DUT

when forwarding broadcast traffic.

91
Layer 2/3/QoS Test Methodology
Forward Pressure Test

„ This test overloads a DUT/SUT port and measures the

output for forward pressure.
„ If the DUT/SUT transmits frames with an interframe gap less
than 96 bits then forward pressure is detected.

92
Layer 2/3/QoS Test Methodology
Measuring the Performance of the
Switching Fabric
„ The ultimate test of the fabric is to load it across all ports
simultaneously.
„ Traditionally, three types of fabrics: shared bus, shared
memory, and crossbar architectures.
„ Today, switching fabrics are Proprietary and Complex.

Switch Fabric

93
Layer 2/3/QoS Test Methodology
VLAN Testing

„ RFCs that describe the Terminology/Methodology for testing

a VLAN switch are:
‹ RFC 1242, RFC 2544, RFC 2285, RFC 2889
‹ No unique RFCs associated for testing VLANs

QoS

L4-7
Bridge L2 Switch
VLAN L3
Router Switch Switch

VPNs

Multicast
IP

94
Layer 2/3/QoS Test Methodology
VLAN Testing Requirements
„ Traffic containment
„ Traffic leakage
„ VLAN recognition
„ Tag imposition
„ Tag removal
„ Maximum frame size
„ Access vs. Trunk frame size
„ Tag field variations
„ Maximum # supported
„ “Core” RFC tests within a multiple VLAN environment
95
Layer 2/3/QoS Test Methodology
Layer 3 Switch Testing
„ RFCs that describe the Terminology/Methodology for testing
a Layer 3 switch are:
‹ RFC 1242, RFC 2544, RFC 2285, RFC 2889
‹ RFC 3222: Terminology for Forwarding Information Base (FIB)
based Router Performance

QoS

L4-7
Bridge L2 Switch
VLAN L3
Router Switch Switch

VPNs

Multicast
IP

96
Layer 2/3/QoS Test Methodology
RFC 3222 Overview

„ RFC 3222, "Terminology for Forwarding Information Base

(FIB) based Router Performance.“
„ Defines terms that are to be used in a methodology that
determines the IP packet forwarding performance of IP
routers as a function of the FIB installed within the router.
„ The objective of this methodology is to evaluate the
performance levels of IP routers as forwarding information
bases continue to grow in size and complexity of structure.
„ This methodology utilizes the packet forwarding
performance measurements described in RFC 2544 and
reference will also be made to the associated terminology
document RFC 1242 for these terms.

97
Layer 2/3/QoS Test Methodology
IP Testing Requirements
„ IP Control Plane Tests:
‹ ARP
‹ PING
‹ SNMP
‹ Routing
‹ ICMP
‹ IGMP
„ IP Data Plane Tests
‹ Forwarding
‹ Errored Packets
‹ Fragmentation
‹ TTL Expired
‹ QoS
‹ Varying Addresses and Prefix Lengths
98
Layer 2/3/QoS Test Methodology
Testing Layer 3 Forwarding
Capabilities
„ Ability of the router (Layer 3 switch) to process incoming packets
‹ Verify the IP header checksum and TTL
‹ Decrement the TTL and recalculate the IP header checksum
„ Look up the destination address within the routing table
„ Forward the packet through the switch fabric
„ Get the packet through the output queues
„ Re-encapsulate the appropriate link layer attributes
Know:
See: Packet Forget:
Packet In Processing Packet Out

99
Layer 2/3/QoS Test Methodology
Logical Model of a Multi-Layer Switch

QoS
Multicast IP Routing function
Management

Layer 2 Layer 2
Switching Switching
Function Function

VLAN 1 VLAN 2 VLAN 3

100
Layer 2/3/QoS Test Methodology
Physical Model a Multi-Layer Switch
„ Most device architectures adopt the same functional components.
„ Each component in a generic multi-layer switch is shown below:

Switch Processor
RIB

Line Card Line Card

Processor Processor

Switch Fabric
Queues Queues

FIB FIB

101
Layer 2/3/QoS Test Methodology
Critical Router Loop

„ The set of instructions that must be executed to determine

how to forward a packet.
„ Each instruction is a different process that you can exercise
while testing the performance of the device.
‹ A Conformance test would determine if the device responds
appropriately under varying conditions taking the entire suite under
consideration.
‹ A Functional test would determine if your router can do a particular
instruction.
‹ A Performance test would determine the effect a particular
instruction has on throughput.

102
Layer 2/3/QoS Test Methodology
IPv4 Critical Router Loop
1. Verify the IP Header Checksum field.
2. Verify that the value of the Version field is 4.
3. Decrement the value of the TTL field.
‹ If its new value is less than 1, send an ICMPv4 Time to Live Exceeded
message to the source
4. Check for the presence of IPv4 header options and if present, process them.
5. Use the Destination Address field to determine a forwarding interface and a
next-hop.
‹ If a route is not found, send an ICMPv4 Destination Unreachable message
to the source.
6. Potentially perform IPv4 fragmentation.
‹ Or if the DF flag is set to 1, send an ICMPv4 Destination Unreachable-
Fragmentation Needed And DF Set message to the source
7. Recalculate the new header checksum and place its new value in the Header
Checksum field.
8. Forward the packet by using the appropriate forwarding interface.
103
Layer 2/3/QoS Test Methodology
Multicast IP Testing
„ RFCs that describe the Terminology/Methodology for testing
Multicast IP are:
‹ RFC 1242, RFC 2544, RFC 2285, RFC 2889
‹ RFC 2432: Terminology for IP Multicast Benchmarking
‹ RFC 3918: Methodology for IP Multicast Benchmarking

QoS

L4-7
Bridge L2 Switch
VLAN L3
Router Switch Switch

VPNs

Multicast
IP

104
Layer 2/3/QoS Test Methodology
RFC 2432 Overview

„ Defines terms used in describing IP multicast performance

benchmarking tests and the results of such tests.
„ Extends terminology/methodology already defined for
benchmarking network interconnect devices in RFCs 1242,
2285, 2544, and 2889.
„ Some of the terms defined by RFC 2432 are:
– Traffic Class (TC)
– Group Class (GC)
– Service Class (SC)
– Multicast Latency (ML)
– Min/Max Multicast Latency (Min/Max ML)
– Group Join Delay (GJD)
– Group Leave Delay (GLD)

105
Layer 2/3/QoS Test Methodology
RFC 3918 Overview
„ Provides the methodology for the benchmarking of local multicast IP
forwarding devices.
„ It builds upon the tenets set forth in RFC 2544, RFC 2432 and other
IETF Benchmarking Methodology Working Group (BMWG) efforts
„ This document seeks to extend these efforts to the multicast paradigm.
„ Defines tests for measuring and reporting the throughput, forwarding,
latency and Internet Group Management Protocol (IGMP) group
membership characteristics of devices that support IP multicast
protocols.
„ The results of these tests will provide the user with meaningful data on
multicast performance.
„ Subsequent documents may address IPv6 multicast and related
multicast routing protocol performance.

106
Layer 2/3/QoS Test Methodology
RFC 3918 Benchmarking Tests
„ Mixed class throughput – router forwarding performance in the
presence of mixed unicast and multicast traffic
„ Aggregate multicast throughput – multicast data throughput as a
function of the number of destination ports or interfaces.
„ Scaled group forwarding – multicast data forwarding performance
as a function of the number of multicast groups.
„ Multicast group capacity – the maximum number of multicast
groups the DUT can support.
„ Multicast latency – data latency applied across a number of output
ports.
„ Group Join/Leave Latency – relates to the characterization of the
overhead delays associated with explicit operations found in
multicast environment.
107
Layer 2/3/QoS Test Methodology
QoS Testing
„ RFCs that describe the Terminology/Methodology for testing
Quality of Service (QoS) device are:
‹ RFC 1242, RFC 2544, RFC 2285, RFC 2889
‹ RFC 4689 Terminology for Benchmarking Network-layer Traffic
Control Mechanisms
‹ RFC 4883 Benchmarking Terminology for Resource Reservation
Capable Routers
QoS

L4-7
Bridge L2 Switch
VLAN L3
Router Switch Switch

VPNs

Multicast
IP 108
Layer 2/3/QoS Test Methodology
RFC 4689 QoS Terminology

„ Terminology for Benchmarking Network-layer Traffic Control

Mechanisms
„ This document introduces several new terms that will allow
measurements to be taken during periods of congestion.
„ Another key difference from existing terminology is the
definition of measurements as observed on egress as well
as ingress of a device/system under test.
„ The principal measurements introduced in this document are
vectors for rate, delay, and jitter.
„ RFC 1242 and RFC 2285 should be consulted before
attempting to make use of this document.

109
Layer 2/3/QoS Test Methodology
RFC 4689 QoS Terminology cont.
„ Configuration Terms
‹ Classification
‹ Codepoint Set
‹ Forwarding Congestion
‹ Congestion Management
‹ Flow
Layer 2/3/QoS Test Methodology
„ Measurement Terms
‹ Channel Capacity
‹ Conforming Packet
‹ Nonconforming Packet
‹ Forwarding Delay
‹ Jitter
‹ Undifferentiated Response
‹ Sequence Tracking
‹ Stream
‹ Test Sequence number
‹ Vectors
110
Testing QoS

„ Deliberately overload an output port.

„ Test the discard policy of the device based on QoS.
„ Test the latency variation of the flows based on QoS.

Congested port

High priority – Telnet

Medium priority – FTP

Low priority – HTTP

111
Layer 2/3/QoS Test Methodology
Frame Formats: QoS Control
Mechanisms
„ Spirent test equipment enables direct access to the Layers 1-4 QoS control
mechanisms using the Flow Attributes tabs:

Layer 1 – Physical Ports.

Layer 2 – MAC Addresses, Type Field, 802.1Q tag (802.1p Priority*).
Layer 3 – IP Addresses, Diffserv* (Type of Service*), Protocol.
Layer 4 – TCP/UDP Ports.
* Indicates an explicit QoS control mechanism.

MAC MAC IP Header TCP/UDP Header

Type Application Data
Destination Source Diffserv, Protocol Source & Destination FCS
Field from a Single User Session
Address Address IP Addresses TCP/UDP Ports

0x8100 Priority CFI VLAN ID

16 3 1 12
112
Layer 2/3/QoS Test Methodology
QoS Testing Requirements
„ Connectionless vs. Connection Oriented Techniques
„ Queuing Techniques (Scheduling vs. Management)
„ “Threading” the Layers
„ Traffic Differentiation (flow recognition)
„ Marking/remarking packets
„ Throughput vs. Packet Loss
„ Latency (Transfer Delay vs. Delay Variation)
„ Sequence Tracking
„ Congestion control and Avoidance
„ Traffic Engineering
113
Layer 2/3/QoS Test Methodology
Additional BMWG RFCs

„ RFC 4883 Benchmarking Terminology for Resource

Reservation Capable Routers
‹ Deals with Integrated Services (i.e., RSVP)

„ RFC 4814 Hash and Stuffing Overlooked Factors in Network

Device Benchmarking
‹ Current benchmarking practice overlooks two factors that have a
profound impact on test results:
• First, existing methodologies do not require the reporting of addresses
or other test traffic contents, even though these fields can affect test
results.
• Second, "stuff" bits and bytes inserted in test traffic by some link-layer
technologies add significant and variable overhead, which in turn affects
test results.

114
Layer 2/3/QoS Test Methodology
End of Topic

www.spirentcampus.com
115
Layer 2/3/QoS Test Methodology