September 2019

#219130

Commissioned by
Mellanox Technologies

Mellanox ConnectX®-5 25GbE Ethernet Adapter


Adapter Performance vs Broadcom NetXtreme® E

EXECUTIVE SUMMARY THE BOTTOM LINE

Data center computing environments for Cloud, Edge Computing and Hyper-
converged infrastructures (HCI) demand high performance throughput, 25GbE or The Mellanox ConnectX-5 25GbE adapter delivered:
higher server connectivity, scalability and resilience.
1 Higher and consistent performance, both RoCE and
Mellanox commissioned Tolly to benchmark the performance of its ConnectX-5 DPDK, at all packet sizes
25GbE network adapter against the performance of Broadcom’s NetXtreme E
2 High scalability, whereas Broadcom connections
25GbE network adapter in cloud and storage computing environments. The series
were limited
of datacenter-oriented tests used included RoCE fabric, FIO, DPDK RFC 2544 and
DPDK CPU throughput. 3 Fabric-agnostic RoCE, including Zero Touch RoCE for
lossy fabrics
The Mellanox ConnectX-5 25GbE adapter demonstrated higher throughput, better
scale and lower resource utilization across the set of test scenarios. 4 Efficiency, dramatically lower server resource
demand than Broadcom

RoCE Throughput & Scale - 25GbE, All to All Scenario

Mellanox ConnectX-5 vs. Broadcom NetXtreme E Mellanox ConnectX-5 - Different Fabric Congestion Scenarios

40 40
Aggregate Throughput (Gbps)

20 20
Broadcom
could not
complete the
test of 2,048
connections

0 0 ECN+PFC ECN, No PFC No ECN, No PFC

64 256 2,048
Number of Connections (Queue Pairs) 64 Connections 2,048 Connections
Mellanox ConnectX-5
Broadcom NetXtreme E
Notes: Broadcom could not complete the test of 2,048 connections. 
 Notes: Broadcom requested that it only be tested with Explicit Congestion
ECN (Explicit Congestion Notification) and PFC (Priority Flow Control) enabled. Notification (ECN) and Priority Flow Control (PFC) enabled.
Source: Tolly, July 2019 Figures 1 and 2

© 2019 Tolly Enterprises, LLC Tolly.com Page 1 of 8

 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance #219130

Test Results RoCE Tests Mellanox, Inc.

Four sets of tests were run using dual-port, The RoCE tests were run using loads of 64,
25GbE adapters and up to seven server 256 and 2,048 connections (also referred to ConnectX-5
systems. Multiple tests were run within as queue pairs or QPs). The switching fabric 25GbE Adapter
each category. Test groups were: was configured different ways: 1) Lossless
Fabric with Explicit Congestion Notification Performance,
RDMA over Converged Ethernet (ECN) and Priority Flow Control (PFC)
throughput and scale tests. RoCE is a high enabled, 2) ECN only enabled, and 3) Lossy Scale & CPU Tested

performance low latency protocol. Fabric with neither ECN nor PFC enabled. Efficiency July

2019
FIO which is a standard storage benchmark, Lossless Fabric
utilizing RoCE capabilities, to evaluate
networked storage throughput. Mellanox outperformed Broadcom at all
deployments, the Broadcom NIC was
three connection levels. At 64 connections,
unable to sustain the throughput. See
DPDK RFC2544 zero-loss packet tests were the Mellanox throughput was 10% greater
Figure 1.
used to measure throughput at different and at 256 connections was more than 2X
standard packet sizes. that of Broadcom. The Broadcom adapters
could not complete the test of 2,048
Partial & Lossy Fabric
TestPMD from the Linux Foundation’s Data connections where Mellanox delivered Mellanox believes that it is important for
Plane Developers Kit (DPDK) was used to 23Gbps of aggregated throughput. As the NICs to be able to deliver reliable
evaluate the CPU efficiency of each number of connections increases, which is throughput in all network configurations
network interface card (NIC). a typical demand for cloud and hyperscale even when a switch fabric is not fully

25GbE Performance: 64 Connections All2All RoCE Throughput 


Mellanox ConnectX-5 vs. Broadcom NetXtreme E

 !+
 



 !+

 
 
"#


(+

,&&&&&
', ')& (+) *.

', ')& (+) *.

'+ ')& (+) *,

(&
', ')& (+) *(
+&&&&&
', ')& (+) *'

 

 

'+ ')& (+) *-
'+ ')& (+) *(

*&&&&&
'+ ')& (+) *'






'+
'+ ')& (+) *(
'+ ')& (+) */

', ')& (+) **
)&&&&&
', ')& (+) *,

'&
', ')& (+) *-

', ')& (+) */

', ')& (+) */

(&&&&&

'+ ')& (+) *)
'+ ')& (+) *.

+
'+ ')& (+) **

'+ ')& (+) **
'&&&&&
', ')& (+) *)

', ')& (+) *'
'+ ')& (+) *-

&

&
', ')& (+) *(

'
)
+
-
/
''
')
'+
'-
'/
('
()
(+
(-
(/
)'
))
)+
)-
)/
*'
*)
*+
*-
*/
+'
+)
++
+-
+/
,'
,)
,+
,-
'+ ')& (+) *,

'
*
-
'&
')
',
'/
((
(+
(.
)'
)*
)-
*&
*)
*,
*/
+(
++
+.
,'
,*
,-
', ')& (+) **

', ')& (+) *)
'+ ')& (+) *)









 


 


 



 
 
"#


,&&&&&

') ')& (+) *.

') ')& (+) *'

(+
+&&&&&
'( ')& (+) **

'( ')& (+) */

') ')& (+) *,
'( ')& (+) *)






*&&&&&

 

 

(&
') ')& (+) *'
'( ')& (+) *-

'( ')& (+) **
') ')& (+) *)

') ')& (+) *-
'( ')& (+) *.

'+
)&&&&&

'( ')& (+) *.

'( ')& (+) *'

'( ')& (+) *(

'( ')& (+) *,
(&&&&&
'( ')& (+) *,

'&

') ')& (+) */
') ')& (+) */

'( ')& (+) *)
'( ')& (+) *(

+
'&&&&&
') ')& (+) **

') ')& (+) *.

') ')& (+) *)
'( ')& (+) */

&
'( ')& (+) *'
&
') ')& (+) *-

') ')& (+) *(
'
*
-
'&
')
',
'/
((
(+
(.
)'
)*
)-
*&
*)
*,
*/
+(
++
+.
,'
,*
,-
') ')& (+) *(

'
)
+
-
/
''
')
'+
'-
'/
('
()
(+
(-
(/
)'
))
)+
)-
)/
*'
*)
*+
*-
*/
+'
+)
++
+-
+/
,'
,)
,+
,-

') ')& (+) **
') ')& (+) *,




'( ')& (+) *-




Source: Tolly, July 2019 Figure 3

© 2019 Tolly Enterprises, LLC Tolly.com Page 2 of 8

 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance #219130

configured to respond to congestion FIO Write Throughput & Scale - 25GbE

control notifications. Broadcom noted that Mellanox ConnectX-5 vs. Broadcom NetXtreme E
such conditions are not covered by 40
standards groups and requested not to be
tested in these scenarios.

Aggregate Throughput (Gbps)

The Mellanox-only results are found in 30
Figure 2. In tests of both 64 and 2,048
connections, the Mellanox solution
delivered virtually identical results without
20
regard to the congestion control state of
the underlying switching fabric.
Broadcom
could not
The aggregate throughput with 64 10 complete
connections was 38Gbps in the first two the test of
scenarios and 37Gbps in the third scenario. 50 jobs.
The aggregate throughput with 2,048
connections remained at 23Gbps across all 0
1 12 50
three scenarios. Mellanox Zero Touch RoCE
Concurrent Jobs
(ZTR) over lossy networks required zero Mellanox ConnectX-5
tuning or configuration of the fabric. ZTR Broadcom NetXtreme E
capability proved able to deliver similar Note: As reported by FIO. Dual-port NICs. Many to one.
throughput and scale as RoCE for simple, Source: Tolly, July 2019 Figure 4
lossy networks.

Performance Consistency DPDK RFC2544 Zero Packet Loss Throughput - 25GbE

Mellanox ConnectX-5 vs. Broadcom NetXtreme E
Figure 3 shows the 64 connection results
100
for both solutions across the 60 second run-
time of the test. The graphs on the left
depict the received throughput and the
graphs on the right depict the number of 75
2.5X
Line Rate (%)

Congestion Notification Packets (CNPs) sent

by each solution.
50
Mellanox ConnectX-5 shows better and
more consistent results over time and the
number of CNPs peaks at about 50,000. By 25
contrast the throughput of the Broadcom
solution varies over time. The number of
CNPs peaks at about 600,000 with the CNPs 0
in the range of 300,000 for most of the test 64 128 256 512 1024 1280 1518
run. Broadcom’s much higher number of Packet Size (Bytes)
CNPs translates to less efficient use of the
Mellanox ConnectX-5
network. Broadcom NetXtreme E
Note: As reported by Ixia IxNetwork. Dual-port NICs.
Source: Tolly, July 2019 Figure 5

© 2019 Tolly Enterprises, LLC Tolly.com Page 3 of 8

 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance
FIO Tests
These storage benchmarking tests
demonstrated the many-to-one networked
“write” performance of the NICs under test
at three increasing load levels.
The Mellanox performance was higher at
all three load levels. See Figure 4. The
Mellanox aggregate throughput for a
single job was 33Gbps - more than 50%
greater than Broadcom’s 21Gbps.
With 12 concurrent jobs, the Mellanox
aggregate throughput was 36Gbps which
was 20% greater than Broadcom’s 30Gbps.
With 50 concurrent jobs, the Mellanox
aggregate throughput was 35Gbps.
Engineers were unable to complete the 50
job test with the Broadcom solution. After
testing was completed and Tolly shared the
results with Broadcom, the vendor noted
that they could not reproduce the problem
with their current software version.
DPDK RFC2544 Zero-Loss
Packet Throughput Tests
Number of CPU Cycles
DPDK consists of libraries to accelerate
packet processing workloads running on a
wide variety of CPU architectures. Run on a
single system outfitted with a dual-port
25GbE NIC, this test illustrated the lossless
throughput of the NIC at the standard
RFC2544 packet sizes that ranged from 64
to 1518 bytes.
The Mellanox solution achieved 100% line-
rate with zero loss at every packet size
tested. The Broadcom solution achieved
wire-speed throughput only at packet sizes
of 1024-bytes and higher. See Figure 5.
At 64-bytes, the line-rate throughput of
Mellanox was 2.5X that of the 40% of line-
© 2019 Tolly Enterprises, LLC
#219130
rate throughput for Broadcom. At 128 Mellanox needed only 49 CPU cycles in the
through 512-byte packets, Broadcom single and dual-core tests and only 59
throughput delivered 70, 71 and 73% of cycles in the quad-core test. By contrast,
line-rate respectively. Broadcom required 534 cycles for the
single-core test, 838 cycles for the dual-core
DPDK CPU Cycles Per Packet and 907 cycles for the quad-core.
Broadcom’s CPU demand was 15X that of
The final test was run on a single system Mellanox for the quad-core test.
and was used to gauge the CPU efficiency
of the NIC under test. The TestPMD utility
was used generate a stream of 1518-byte
Test Setup &
packets and measure the number of CPU
cycles required to process a single packet. Methodology
The test was run using single, dual and
Details of server configuration, SUT
quad cores active in the server.
configurations, release levels and relevant
The Mellanox solution demonstrated better information are found in Tables 1-3.
CPU efficiency in all tests with dramatically
lower CPU requirements. See Figure 6.
DPDK CPU Cycles Per 1518-Byte Packet - 25GbE
Mellanox ConnectX-5 vs. Broadcom NetXtreme E

(Lower numbers are better)
1000
750
15X
500
250
0
Single-core Dual-core Quad-core
Number of CPUs
Mellanox ConnectX-5
Broadcom NetXtreme E
Note: As reported by TestPMD. Dual-port NICs.
Source: Tolly, July 2019 Figure 6
Tolly.com Page 4 of 8
 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance
RoCE Fabric Tests
This benchmark tested an All to all (A2A)
scenario with all servers both transmitting
and receiving traffic.
Seven hosts (servers) were used for these
tests. Each host was outfitted with dual-
port 25GbE NICs from Mellanox and
Broadcom. All ports were connected to a
Mellanox Spectrum Ethernet switch.
Testing was conducted using Red Hat
Enterprise Linux (RHEL) 7.4 and respective
SUT driver and firmware.
Tests included three different fabric
configurations:
1. FlowControl=ON, ECN=ON
2. FlowControl=OFF, ECN=ON
3. FlowControl=OFF, ECN=OFF
The Mellanox NIC was tested in all three
configurations. At Broadcom’s request, their
NIC was only tested in the first
configuration.
Fabric Configuration Steps
RDMA traffic was classified into traffic-class
3 via DSCP 0x18. Congestion notification
packets (CNPs) were assigned to traffic-
class 6 via DSCP 0x30.
Flow control was configured as required on
both the switch and NICs. When ECN was
enabled, ECN on the switch was configured
via specifying a separate “lossless” buffer for
traffic-class 3. ECN on Mellanox NICs is
auto-configured, engineers specified DSCP-
based classification via ‘mlnx_qos’. ECN on
Broadcom NICs was configured via the '/
sys/kernel/config/bnxt_re/$INTF/ports/1/
cc' interface.
© 2019 Tolly Enterprises, LLC
Test Procedure
Tests were run as follows:
• On each host, for each NIC port, 6
ib_write_bw server processes were open
(1 for each of the other hosts)
• On each host, for each NIC port, 6
ib_write_bw client processes were open
(1 sending traffic to the respective port of
each of the other hosts)
Engineers tested 64 QPs per ib_write_bw
process, 256 QPs per ib_write_bw process,
and 2048 QPs per ib_write_bw process.
Engineers forced all RDMA traffic to be sent
with DSCP 0x18 via ib_write_bw '--
tclass=96' parameter.
Message size was 64KB (RDMA Write data
size).
The test result was the average traffic per
host: sum of ib_write_bw client results on a
host (both NIC ports), averaged across the 7
hosts. Test duration was 60 seconds and
repeated three times. Results were
averaged.
FIO NVMe-oF
This test focused on the FIO throughput of
a single host, the NVMe-oF Target.
Test Equipment Summary
The Tolly Group gratefully acknowledges the providers
of test equipment/software used in this project.
Vendor Product
Optixia XM12

Ixia Software: IxNetwork 8.5

Tolly.com
#219130
Seven hosts (servers) were used for these
tests. Each hosts was outfitted with a dual-
port 25GbE NIC from Mellanox and
Broadcom. All ports were connected to a
Mellanox Spectrum Ethernet switch.
Testing was conducted using Red Hat
Enterprise Linux (RHEL) 7.6 and respective
SUT driver and firmware.
Fabric configuration testing was the same
as for the RoCE fabric tests.
Test Procedure
Target configuration steps overview:
• Load 'nvmet-rdma' kernel module
• Create null block devices: 'modprobe
null_blk nr_devices=2'
• Create an nvmet_rdma subsystem
connected to /dev/nullb0, link it and start
listening on NIC port0
• Create an nvmet_rdma subsystem
connected to /dev/nullb1, link it and start
listening on NIC port1.
Initiator configuration steps (run on six
systems):
• load 'nvme-rdma' kernel module
• Connect to the Target via 'nvme-
cli' ('discover' via Target IP and port, the
'connect')


Web


http://www.ixiacom.com
Page 5 of 8
 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance #219130

25 Gigabit Ethernet: Dual-Port Adapters Under Test

Vendor Product Name Part No. Driver Driver Version Firmware Version

Mellanox ConnectX-5 CX512A MCX512A- mlx5_core MLNX_OFED_LINUX-4.6-1.0.1.1 16.25.1020

ACAT

NetXtreme-E 817716-001 bnxt_en, 1.9.2-214.0.182.0 (NVMe-oF test) 214.0.202.0/pkg 214.0.203000

Broadcom
(HPE) BCM57414 B1 bnxt_re 1.9.1-212.0.99.0 (Write BW test) 212.0.102/1.9.1/pkg 212.0.1030
(631SFP28)
Table 1

Test Infrastructure

RDMA Write BW & NVMe-oF RoCE FIO Read/Write DPDK

Number of Servers 7 Number of Servers 1

Make & Model Dell PowerEdge R630 Make & Model HPE ProLiant DL380 Gen10
(868703-B21)
CPU 2 * Intel(R) Xeon(R) CPU E5-2660 v4 @
2.0GHz, 14 physical cores per socket, 2 * Intel(R) Xeon(R) Gold 5118
HyperThreading=ON
CPU
CPU @ 2.30GHz, 12 physical
cores per socket,
Memory (RAM) 32GB per NUMA node, 16GB DIMMS DDR4
HyperThreading=ON
@ 2400 MT/s Dual Rank

Disk Controller HBA330 12Gbps

Memory (RAM) 12 * 16GB DIMMS @ 2666 MT/s
HDD Seagate 600GB ST600MM0088 Dual Rank (96GB per NUMA
socket)

Integrated RAID PERC H730P Mini

Server OS FIO: Red Hat Enterprise Linux
Controller Server 7.5

Server OS FIO: Red Hat Enterprise Linux Server 7.6 Add’l software for libbnxt_re-214.0.181.0-
RoCE: RHELS 7.4 Broadcom rhel7u6.x86_64.rpm

Add’l software for RDMA: libbnxt_re-212.0.82.0- IxNetwork 8.51

rhel7u4.x86_64.rpm
Traffic Generator
Broadcom Software
FIO: libbnxt_re-214.0.181.0-
ib_write_bw (part of 5.60 DPDK 19.02
perftest, packaged with
MLNX OFED)
Traffic Generator Ixia XM12
Fio 2.1.10

Switches 100GbE with 4*25GbE Splitters

Table 2 Table 3
Source: Tolly, June 2019

© 2019 Tolly Enterprises, LLC Tolly.com Page 6 of 8

 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance
NVMe-oF devices ‘/dev/nvme0n0’ ‘/dev/
nvme0n1’ then appear on each Initiator (1
for each NIC port).
FIO storage benchmark is run on each
Initiator, to issue read/write requests to ‘/
dev/nvme0n0’, ‘/dev/nvme0n1’.
NVMEoF transports the read/write requests
via RDMA to the Target.
FIO Parameters
• job composition "--rw":
• 'randread' = the workload consists of
100% read requests, to random
locations in the destination file
• 'randwrite' = the worload consists of
100% write requests, to random
locations in the destination file
• number of FIO processes "--numjobs": 1,
12, or 50
• blocksize: 4KB
• runtime: 70 sec. "--ramp_time": 5 sec.
The test results are: 1) the sum of the FIO
reported BW on all the Initiators (this sum is
limited by Target's NIC TX/RX on both ports
= 50GbE) and 2) the sum of the FIO
reported IOPS on all the Initiators (this sum
is limited by the target's NIC TX/RX on both
ports = 50GbE),
DPDK Tests
Data Plane Development Kit (DPDK)
consists of libraries to accelerate packet
processing according to its source, The
Linux Foundation. These are low level,
hence technical, tests. Thus, the test
description is necessarily very granular.
Two tests were run on DPDK. L3fwd was
run to find the maximum rate of
© 2019 Tolly Enterprises, LLC
throughput for standard RFC2544 packet
sizes with maximum frame loss of 0.001%.
TestPMD was run against 100% line rate
traffic to determine the number of host
CPU cycles required to process a given
frame. Tests were conducted on RHEL 7.5.
DPDK 19.02 was used for testing.
TestPMD Procedure
• Install clean Red Hat Enterprise Linux 7.5
• Install MLNX_OFED_LINUX-4.6-0.3.1.0
• Enable mlx5 PMD before compiling
DPDK:
• In .config file generated by "make config”
• set: “CONFIG_RTE_LIBRTE_MLX5_PMD=y"
• set:
“CONFIG_RTE_TEST_PMD_RECORD_CORE
_CYCLES=y"
• Compile DPDK
TestPMD Optimization
During testing, TestPMD was given real-
time scheduling priority. An Ixia traffic
generator was used to generate 100% line-
rate 1518-byte frames. Only TestPMD
cycles-per-frame results were recorded as
packet throughput/loss was not relevant in
this scenario.
(For Mellanox only):   *Note: For Broadcom,
Flow Control was disabled via TestPMD
runtime command.
a) Flow Control OFF:
"ethtool -A $netdev rx off tx off"
(For System):
b) Memory optimizations:
"sysctl -w vm.zone_reclaim_mode=0";
"sysctl -w vm.swappiness=0"
Tolly.com
#219130
c) Move all IRQs to far NUMA node:          
"IRQBALANCE_BANNED_CPUS=$LOCAL_N
UMA_CPUMAP irqbalance - -oneshot"
d) Disable irqbalance:
"systemctl stop irqbalance"
(For Mellanox only):  
e) Change PCI MaxReadReq to 1024B for
each port of each NIC:
Run "setpci -s $PORT_PCI_ADDRESS 68.w",
it will return 4 digits ABCD -->
Run "setpci -s $PORT_PCI_ADDRESS
68.w=3BCD"
f) Set CQE COMPRESSION to “AGGRESSIVE”:
mlxconfig -d $PORT_PCI_ADDRESS set
CQE_COMPRESSION=1
(For System):
g) Disable Linux realtime throttling:
echo -1 > /proc/sys/kernel/
sched_rt_runtime_us
L3fwd Procedure
An Ixia traffic generator was used to
generate layer 3 (L3) traffic according to
IETF RFC2544.
Traffic with Ethernet and IP headers, with
8K different srcIP addresses, was sent from
each of the two Ixia ports.
Frame sizes were: 64, 128, 256, 512, 1024,
1280, and 1518-bytes.
Page 7 of 8
 Broadcom
Mellanox
NetXtreme
ConnectX-5
25GbE25GbE
vSANPerformance
Performance #219130

About Tolly Interaction with Competitors

The Tolly Group companies have been
delivering world-class IT services for 30
years. Tolly is a leading global provider of
third-party validation services for vendors of
IT products, components and services.
In accordance with Tolly’s Fair Testing Charter, Tolly personnel invited
representatives from Broadcom to view the test details and their results.
Broadcom responded and was provided with their results prior to
publication. See report for relevant comments.

You can reach the company by E-mail at

sales@tolly.com, or by telephone at
 For more information on the 

+1 561.391.5610. Tolly Fair Testing Charter, visit:
Visit Tolly on the Internet at:

http://reports.tolly.com/FTC.aspx
http://www.tolly.com

Terms of Usage
This document is provided, free-of-charge, to help you understand whether a given product, technology or service merits additional
investigation for your particular needs. Any decision to purchase a product must be based on your own assessment of suitability
based on your needs. The document should never be used as a substitute for advice from a qualified IT or business professional. This
evaluation was focused on illustrating specific features and/or performance of the product(s) and was conducted under controlled,
laboratory conditions. Certain tests may have been tailored to reflect performance under ideal conditions; performance may vary
under real-world conditions. Users should run tests based on their own real-world scenarios to validate performance for their own
networks.
Reasonable efforts were made to ensure the accuracy of the data contained herein but errors and/or oversights can occur. The test/
audit documented herein may also rely on various test tools the accuracy of which is beyond our control. Furthermore, the
document relies on certain representations by the sponsor that are beyond our control to verify. Among these is that the software/
hardware tested is production or production track and is, or will be, available in equivalent or better form to commercial customers.
Accordingly, this document is provided "as is," and Tolly Enterprises, LLC (Tolly) gives no warranty, representation or undertaking,
whether express or implied, and accepts no legal responsibility, whether direct or indirect, for the accuracy, completeness, usefulness
or suitability of any information contained herein. By reviewing this document, you agree that your use of any information contained
herein is at your own risk, and you accept all risks and responsibility for losses, damages, costs and other consequences resulting
directly or indirectly from any information or material available on it. Tolly is not responsible for, and you agree to hold Tolly and its
related affiliates harmless from any loss, harm, injury or damage resulting from or arising out of your use of or reliance on any of the
information provided herein.
Tolly makes no claim as to whether any product or company described herein is suitable for investment. You should obtain your own
independent professional advice, whether legal, accounting or otherwise, before proceeding with any investment or project related
to any information, products or companies described herein. When foreign translations exist, the English document is considered
authoritative. To assure accuracy, only use documents downloaded directly from Tolly.com.  No part of any document may be
reproduced, in whole or in part, without the specific written permission of Tolly. All trademarks used in the document are owned by
their respective owners. You agree not to use any trademark in or as the whole or part of your own trademarks in connection with
any activities, products or services which are not ours, or in a manner which may be confusing, misleading or deceptive or in a
manner that disparages us or our information, projects or developments.

219130 nfm-5 -wt 2019-09-22 - VerK

© 2019 Tolly Enterprises, LLC Tolly.com Page 8 of 8