Professional Documents
Culture Documents
OctoScope SmallNetBuilder's Wireless Testbed
OctoScope SmallNetBuilder's Wireless Testbed
SmallNetBuilder's
Wireless Testbed V2
WI-FI BENCHMARK TESTBED
BY TIM HIGGINS
www.octoscope.com
+1.978.222.3114
Introduction
After spending the past year or so with our octoScopebased wireless testbed, I was ready to listen when
octoScope suggested upgrading.
The V1 testbed uses octoBox 26" wide chambers, which
have proved to be a bit too tight, especially as the size of
routers have grown. The desired 8" distance between
router under test and the chamber antennas has not
been achieved on more than one occasion.
Enter the 38" wide octoBox Stackable anechoic
chamber. It measures 24" x 38.35" x 31.2" vs. 18" x
26.35" x 25.2" for the 26" octoBox. The photo below
shows the new stack, with a new 38" wide octoBox MPE
(Multi Path Emulator) sandwiched between the upper
and lower chambers. The key features of the new
configuration are described in the octoScope video here.
The Stack
The new stack includes
octoScope's new quadAtten
programmable attenuator
module, which replaces the
multiple Vaunix Lab Brick attenuators in the old setup.
Dual-shielded cables are also used externally, providing
better shielding for test environments with a lot of RF
activity. The stack sits on a sturdy base with honkin' big
casters that provide very easy rolling. That's an APC
Back-UPS 550 and TRENDnet TEG-S80g Gigabit switch sitting on
www.octoscope.com
Tel.: +1.978.222.3114
Note that the new testbed can handle four RF paths vs. three in the old testbed. This will let us test the new crop of
4x4 AC2350 routers, like ASUS' RT-AC87 that has been available for just about a week as I write this. The 38"
octoBoxes are much roomier inside as the photo below shows.
www.octoscope.com
Tel.: +1.978.222.3114
Everything is connected as shown in the diagram below. As is our standard, the new process uses Ixia's IxChariot to
generate traffic and measure throughput.
Computers
The V2 testbed has gotten a computer upgrade, too. The Dell Optiplex 790 Small Form Factor (Core i5-2400 @ 3.1
GHz) formerly used in the lower chamber now connects to the upper chamber and runs theIxChariot console and the
IxChariot test endpoint for the Device Under Test (DUT). The Test STA computer is also a Dell Optiplex SFF
machine, but a 9010 model with Core i5 3570 CPU @ 3.4 GHz. After a briefly wrestling with Windows 8 quirks, I
decided to stay with Windows 7 Pro on both systems.
Both computers have a TP-LINK TG-3468 PCIe Gigabit Ethernet adapter (Realtek RTL8168B based) installed to
provide a second port. This adapter replaces the aging Intel CT's in the V1 testbed that were starting to become
unreliable. One port connects to the SNB LAN for control, the other to the DUT or reference STA for test data.
www.octoscope.com
Tel.: +1.978.222.3114
Upper Chamber
The signals from the device under test (DUT) are picked up by four omnidirectional unity-gain 2.4 / 5 GHz dualband antennas, shown on the right in the photo below. Because the test client is still 3x3, only three antennas are
currently used. The fourth signal path is sealed with connector caps on both ends. You can see there is now lots of
room in the cabinet, even with the largest router seen to date, NETGEAR's R8000 Nighthawk X6.
Some readers pointed out that the all-vertical antenna orientation in the V1 testbed might provide a performance
edge to routers with vertical antenna polarization. So in the V2 testbed, only one antenna remains vertical; the other
two are set at 45. The two Ethernet and USB connectors and quad power outlets are all filtered so that RF gets
neither in nor out.
That's a SNB-installed bamboo rod running across the top of the chamber. It is used to hang the power and Ethernet
DUT cables so that they don't get hung up as the turntable rotates the DUT. It's also handy to hang an LED worklight
used when setting up routers for test. octoScope provided a plastic crossbar for this, but it was too tall and barely
missed the top of the ASUS RT-AC68U's antennas.
NETGEAR R8000 router under test inside the upper test chamber
I ran many experiments before settling on the antenna configuration. I found the larger octoBox tended to yield
lower throughput than the smaller box, especially for the 5 GHz band. Experiments showed the lower throughput was
primarily due to higher path loss from the greater distance between the DUT and chamber antennas.
I found the slightly closer antenna spacing helped compensate for some of the distance-caused path loss. The
antenna array is centered on the chamber turntable. Distance from center of turntable to chamber antennas is 18
inches (45.72 cm).
www.octoscope.com
Tel.: +1.978.222.3114
Speaking of which, the octoBox-TT is a new feature in the upper chamber, used to rotate the DUT during testing. Its
exact use is described in How We Test Wireless Products - Revison 8. The center of the turntable is approximately
XX" from the chamber antennas.
Between
The three antennas are connected to octoScope's new quadAtten programmable attenuator used to adjust signal
level to simulate distance. The quadAtten can be seen in the first photo above, mounted on the side of the upper test
chamber. The quadAtten outputs are connected to a SMA connectors on theoctoBox MPE, which sits between the
upper and lower chambers. The MPE emulates multipath using IEEE 802.11n/ac indoor channel models B or C. The
lower set of SMA connectors on the MPE connect directly to the lower test chamber, which holds the
standard wireless test client (STA).
Lower Chamber
The V2 testbed retires the ASUS PCE-AC66 AC1750 class PCI-e board used as the reference client in V1 testbed.
After careful consideration, NETGEAR's R7000 Nighthawk running firmware V1.0.3.60_1.1.27 configured in client
bridge mode was selected as the new testbed reference client.
The R7000 is directly cabled to the MPE using dual-shielded cables. Direct cabling is necessary to ensure high
enough signal levels in both 2.4 and 5 GHz so that maximum link rates can be reached. Since the R7000 is an
www.octoscope.com
Tel.: +1.978.222.3114
AC1900 class router, we now can test the 2.4 GHz side of AC1900 routers to their full potential. The R7000 is
connected via Gigabit Ethernet to the Dell Optiplex. The R7000 is managed via web browser from the lower
chamber computer. The computer is accessed remotely via VNC.
That concludes the tour of the V2 testbed. How We Test Wireless Products - Revison 8 describes how we use it.
www.octoscope.com
Tel.: +1.978.222.3114
Introduction
NOTE:This article describes the wireless test procedure for products tested after July 25, 2014.
For wireless products tested between April 15, 2013 and July 24, 2014, see this article.
We described the technology used in our new wireless test process in this article. Now it's time to get into the exact
details of how we use the new testbed.
Environment
Our V8 process is similar to the V7 process in that it uses octoScope anechoic chambers to provide a repeatable RF
controlled test environment. The V2 testbed used by the V8 process has a turntable, which plays a key role in the
new process.
www.octoscope.com
+1.978.222.3114
The V8 process has been designed to address some weaknesses of the V7 process. Specifically:
www.octoscope.com
Tel.: +1.978.222.3114
Key 5 GHz values are shown in the plot below. Throughput at 0 dB attenuation is again used for Maximum Wireless
Throughput Ranking and throughput at 39 dB attenuation is used for Wireless Range Ranking in
the Router and Wireless product Rankers. Less attenuation is used for range ranking in 5 GHz vs. 2.4 GHz due to
higher path loss at those frequencies.
www.octoscope.com
Tel.: +1.978.222.3114
Test Setup
All wireless router performance testing is done on the LAN side of a router under test so that the product's routing
performance does not affect wireless performance results, i.e. the Ethernet client is connected to a router LAN port.
Access points are tested using their Ethernet port.
The wireless router / AP under test (DUT) is centered on the upper test chamber turntable in both X and Y axes. If the
DUT has external antennas, they are centered on the turntable. If the antennas are internal, the router body is
centered. Distance from center of turntable to chamber antennas is 18 inches (45.72 cm).
Initial orientation (0) is with the DUT front facing the chamber antennas. The photo below shows the 0 starting
position for a NETGEAR R8000 Nighthawk X6 router, which has six external antennas arrayed along the sides of
the product.
www.octoscope.com
Tel.: +1.978.222.3114
And here is an ASUS RT-AC68U in its starting test position. (The white object is an LED work light removed during
testing.) Power and Ethernet cables are routed to the DUT from above, to not interfere with DUT rotation.
www.octoscope.com
Tel.: +1.978.222.3114
Test Process
The general test process is as follows:
1.
2.
3.
4.
5.
6.
7.
Two test runs are made in each band for dual-band devices.
Testing is controlled by a Tcl script modified from a standard script supplied by octoScope. The script executes the
following test plan:
1.
Move the turntable to starting position. This places the DUT at the "0 degree" starting position previously
described.
2.
3.
4.
Start 90 second IxChariot test (throughput.scr with 5,000,000 Byte test file size) simultaneous up and
downlink (0 dB only)
5.
6.
7.
Discard first 30 seconds of IxChariot data, calculate average of remaining data and save to CSV file. Save
entire IxChariot .tst file.
8.
9.
www.octoscope.com
Tel.: +1.978.222.3114
The CSV files from the two test runs are merged into a single Excel file and the average of the two runs calculated.
The average of the two runs is the value entered into the Charts database.
Test Notes
Moving the DUT through an entire 360 while the IxChariot test is running removes any orientation bias from the test
and eliminates the need to determine the "best" run out of multiple fixed-position tests. Because throughput typically
varies during the test run, the averaged result tends to be lower. Rotation at lower signal levels, particularly in 5 GHz,
frequently causes the connection to be dropped sooner than if the DUT were stationary. For this reason, V8 5 GHz
test results will generally not extend beyond 39 dB.
The IxChariot plot below shows a typical test run. The first 30 seconds of the run include a throughput step-up that is
caused by an IxChariot artifact and does not reflect true device performance. The first 30 seconds also provides time
for DUT rate adaptation algorithms to settle. For both reasons, the test script excludes the first 30 seconds of data in
the average value saved in the Router Charts database.
Our testing focus has always been to provide results that provide the fairest relative comparison among products.
This new process achieves that goal by removing multiple sources of test setup bias to achieve even fairer productto-product comparison.
www.octoscope.com
Tel.: +1.978.222.3114