Professional Documents
Culture Documents
SA 101911 Quantifying The User Experience - REPORT PREVIEW
SA 101911 Quantifying The User Experience - REPORT PREVIEW
HS-PDSCH HSSCCH MAC-HS E-DPCH FTP UDP HTTP RTT TCP dBm dB Redefining Research Mbps ms MOS LTE HSPA+ DC-HSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS SNR MIMO RI BLER 64QAM 16QAM QPSK CQI L1 DL RSSI RSRQ RSCP CINR ACK NACK PDSCH HS-PDSCH HS-SCCH MAC-HS E-DPCH FTP UDP HTTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ DCHSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS SNR MIMO RI BLER 64QAM 16QAM QPSK CQI L1 DL RSSI RSRQ RSCP CINR-20ACK NACK PDSCH HS-PDSCH HS-SCCH -30 -40 -50 MAC-HS E-DPCH FTP UDP HTTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ DC-HSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS SNR MIMO RI BLER 64QAM 16QAM QPSK CQI L1 DL RSSI RSRQ RSCP CINR ACK NACK PDSCH HS-PDSCH HS-SCCH MAC-HS E-DPCH FTP UDP HTTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ DC-HSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS SNR MIMO R BLER 64QAM 16QAM QPSK CQI L1 DL RSSI RSRQ RSCP CINR ACK NACK PDSCH HS-PDSCH HS-SCCH MAC-HS E-DPCH FTP UDP HTTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ DC-HSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS SNR MIMO RI BLER 64QAM 16QAM QPSK CQI L1 DL RSSI RSRQ RSCP CINR ACK NACK PDSCH HS-PDSCH HS-SCCH MAC-HS E-DPCH FTP UDP HTTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ DC-HSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS SNR MIMO RI BLER 64QAM 16QAM QPSK CQI L1 DL RSSI RSRQ RSCP CINR ACK NACK How not to spend your summer vacation PDSCH HS-PDSCH HS-SCCH MAC-HS E-DPCH FTP UDP VOLUME 2 PREVIEW EDITION HTTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ DCHSDPA TD-LTE EV-DO Rev A/B Mobile WiMAX MCS MCS
8,000 kbps 7,000
Other Nokia Kyocera Palm LG
100% 90% 80% 70%
Other
HTC
Motorola
Samsung
Apple
Other RIM
6,000 5,000
60% 50%
4,000 3,000
52.9%
35.7%
40% 30%
2,000 1,000 0
PHY Layer Primary Throughput (Kbps) PHY Layer Secondary Throughput (Kbps)
38:09.0 37:09.0 36:45.0 36:57.0 37:45.0 37:57.0 37:33.0 37:21.0
18.5%
36:09.0
35:45.0
36:33.0
35:57.0
36:21.0
38:21.0
17:29.0 17:37.0 17:45.0 17:53.0 18:01.0 18:09.0 18:17.0 18:25.0 18:33.0 18:41.0 18:49.0 18:57.0 19:05.0 19:13.0 19:21.0 19:29.0 19:37.0 19:45.0 19:53.0 20:01.0 20:09.0 20:17.0 20:25.0 20:33.0 20:41.0 20:49.0 20:57.0 21:05.0 21:13.0 21:20.0 21:27.0 21:35.0
InItIal feedback receIved In the fIrst few hours after releasIng volume 1 of our three-part serIes Included the followIng comments:
This is a real differentiatorwell worth it. Marketing Director, infrastructure supplier Very detailed and above and beyond my expectations. CTO, tier two operator Great report as always. CTO, leading mobile operator High impact stuffNobody even comes close to delivering this kind of data. Nice, nice move. Marketing Director, infrastructure supplier YOU kick $@ttThis is great stuff. Managing Director, financial institution Great report again. Senior Technical Fellow, infrastructure supplier Great stuffwell thought out and most of all fair. Network Services, leading mobile operator
This document contains a highly-redacted executive summary, a complete table of contents, and our test methodology (Chapter 8 from the main report) for a signals ahead research product that we published on October 18th. All three reports were done with the support of Accuver, who provided us with access to its complete set of network benchmarking tools and post-processing software. Volume 1 of our special three-part series was published in late September and Volume 3 will be published in November. Additionally, this report preview provides a summary of past topics that we have covered in Signals Ahead and a list of likely topics that we plan to tackle in the coming months. The 66-page report contains 53 figures and tables, with many of the figures and tables consisting of multiple parts. This report can be purchased separately for $1,495 or it is included with any paid corporate subscription to signals ahead.
In Volume 1 (Network and Technology Performance) of our special the leading next-generation mobile broadband wireless technologies performed, as exemplified by deployments in North America. To a large
at all times of the day and night trying to suck as much capacity from the
since no one in their right mind drives around large metropolitan areas
expensive in many markets due to existing rate plans and data usage caps.
what really matters to consumers and [hopefully] operators is the impact of the underlying network/ technology on the typical user experience while consuming mobile data. Instead, what really matters to consumers and [hopefully] operators is the impact of the underlying network/technology on the typical user experience while consuming mobile data. In the infancy of a new commercial network launch, technology neophytes and pundits flock to their speed testing website de jour (e.g., www.speedtest.net) and they bask in the glory of never-before-seen data rates. However, this activity and the use of more sophisticated network benchmark studies quickly give way to more typical user behavior. Long-term sustained FTP downlink transfers get replaced by interactions with video delivery services (Netflix, YouTube, etc), content providers (e.g., Apples iTunes) or service enablers (e.g., Googles email). Uplink FTP transfers give way to more challenging scenarios involving the transmitting of video content from the mobile device to the network (e.g., Skype video). Ongoing 32 byte ping tests to a test server get replaced by basic web browsing of popular mobile web sites where the throughput potential of the network and in-network latency take a backseat to the end-to-end latency of the entire connection and how quickly the mobile device and the host site can communicate with each other and pass data back and forth. In Volume 2 (Quantifying the User Experience), we expose readers to a large number of neverbefore-seen test results that we collected during the data collection portion of this exercise how we spent our summer vacation. Although our conclusions from the first report remain intact, there are several nuances to those observations which need to be considered. To summarize, throughput still matters, but only to a certain point since most applications require only a fraction of the throughput offered by todays mobile broadband networks while in many cases the chokepoint in the broadband connection is the wired Internet/host server and not the operators next-generation mobile broadband network. Latency, including both the time to connect to the network and start receiving data, as well as the subsequent interactions between the device and the network/host server, can never be too low. In that regard, too much attention is paid to the my pipe is bigger than your pipe mantra when the real focus should be on delivering a more compelling user experience, regardless of how it is achieved. Our ability to collect and analyze the user experience data would not have been possible without the support of Accuver, who allowed us to use its suite of network drive test tools, including its recently released XCAL-MO network benchmarking tool and XCAL-M drive test solution, as well as its XCAP post-processing software to analyze the results. We have used the Accuver tools
throughput still matters, but only to a certain point while latency can never be too low.
we were able to capitalize on many of the unique features associated with the suite of accuver tools when we conducted our user experience tests.
operators should re-evaluate how they promote and market their services, starting by replacing the focus on speed with the focus on fast.
several times in the past for various Signals Ahead reports and we have grown quite fond of their capabilities and their ease of use. In particular, we were able to capitalize on many of the unique features associated with the suite of test tools when we did our user experience tests. We look forward to leveraging other innovative capabilities associated with their suite of tools that pertain to quantifying the user experience in the near future. For the first time, we are now able to offer their tools with our services for commissioned-based projects on behalf of operators, government regulators, vendors, trade associations or other interested parties on a global basis. We look forward to discussing such opportunities with anyone that is interested. As we discuss in Chapter 3 of this report, operators should re-evaluate how they promote and market their services. To some extent it is a bit too late since theyve sold consumers on the concept that speed is all that matters. However, it is possible to replace speed with fast, and then equate fast with the time required to connect to the network, as well as the time required to access and/ or download content. As an analogy, operators dont advertise if they use half-rate AMR, full-rate AMR or wideband AMR, but they do advertise voice quality, call completion rates, dropped call rates, etc. In a similar fashion, handset manufacturers dont promote lower current drain; instead they advertise longer battery lives. Those metrics truly define the user experience. In this context, an improved user experience can be achieved with a better handset design and an improved operating system GUI that make it easier to enter commands or navigate the web, just as it can be achieved by improvements to the network. Further, operators do not necessarily need to rush out and deploy the latest technology on the market to make their networks faster. Instead, these operators could take an existing technology and simply make it better and more efficient. This goal could be achieved through a combination of network optimization initiatives, by taking full advantage of the networks capabilities, including the widespread use of handsets that support the same set of robust features (e.g., Enhanced_FACH, CPC, CELL_PCH/URA_PCH, Cat 14, etc), and by moving popular content closer to the edge of the network. Chapter 2 of this report contains the introduction and Chapter 3 contains the key conclusions and observations from our tests. It is by far the most important chapter in this report. The remaining chapters and the appendix document the basis for our conclusions. Chapter 4 presents results from numerous tests involving HTTP sessions where we analyze how long it takes to load a popular web page for a given combination of throughput and latency, not to mention other considerations. Chapter 5 explains why throughput doesnt [always] matter and Chapter 6 explains why latency [almost] always matters. Chapter 7 focuses on bandwidth-intensive applications, such as Gmail, iTunes, Netflix, YouTube and to a lesser extent Skype Video. Chapter 8 provides our test methodology and Chapter 9 provides some closing remarks. The Appendix provides results from a few test scenarios that didnt find their way into the main body of the report. Our three-part series of reports is included with a subscription to Signals Ahead or it can be purchased on an individual basis the former option is far more economical since it includes at least 14 additional Signals Ahead reports. A summary of Volume 1 and Volume 3 follows in the subsequent paragraphs.
volume 1 is critical for operators around the globe who are currently making strategic decisions regarding their network technology evolution.
median and CDF plots comparisons, including versus LTE (2x10MHz) with 2x2 MIMO
Technology
DC-HSDPA LTE
(2x20MHz) versus LTE (2x10MHz) WiMAX versus HSPA+, LTE and DC-HSDPA versus HSPA+ versus LTE, HSPA+, etc normalized for channel bandwidth and duplex scheme
Mobile
DC-HSDPA EV-DO
Single User Spectral Efficiency Results LTE with MIMO really outperform narrow bandwidth solutions network performance with multiple devices and HSPA+ devices in the same 10MHz channel allocations
DC-HSDPA
Side-by-Side operator network coverage maps for drive routes used in each market throughput throughput Latency based on time of day and network loading
Downlink Uplink
Network Variance
LTE network deployment philosophies (LTE cell site density relative to the legacy network) and
their implications for coverage and capacity HSPA+ versus AT&T LTE
Wireless EV-DO versus Verizon Wireless LTE LTE versus Verizon Wireless LTE WiMAX (2500MHz) versus LTE (700MHz)
volume 2 is critical for operators trying to understand how to market their broadband wireless service offering as well as how they should prioritize their network optimization activities.
websites, including Yahoo, CNN, iTunes, Amazon, YouTube, etc. down to the millisecond, based on device/chipset signaling messages comparisons
Comparisons based on network loading same location over a 12-15 hour period of time if perceived differences in network/technology performance have more to do with network loading than the actual capabilities of the network/technology itself how the combination of throughput and latency impact the HTTP web page download time results axis plot, showing maximum achievable throughput and network latency versus webpage load time and required throughput matters most latency or throughput the crossover point when higher throughput become irrelevant the user experience based on downloading Google email attachments the user experience based on downloading video and audio content from iTunes the crossover point when higher throughput become irrelevant DC-HSDPA really offer a quantifiable benefit over HSPA+
Which Does
Determining the chokepoints in the network (from end user to the original source of the content),
how they vary as a function of loading, and their impact on the user experience
volume 3 provides invaluable competitive intelligence while also allowing readers to obtain a greater appreciation for how each technology delivers its results.
Utilization (QPSK, 16QAM, and 64QAM) by primary and/or secondary carriers as appropriate RI 1 and RI 2 how MIMO performs at 700MHz Codes (average, % > 10, distribution) by primary and secondary carriers
CQI (average and median) by primary and secondary carriers Scheduling Success Rate by primary and secondary carriers
PHY Layer Served Rate by primary and secondary carriers PHY Layer Scheduled/Served Rate DC-HSDPA only
Maximum UL
We will also leverage the capabilities of the XCAP-M tool to analyze these KPIs by several different means, potentially including, but not limited to the following:
MAC-HS MAC-HS Reported
Throughput versus RSCP scatter plot Throughput versus Reported CQI Values scatter plot Throughput versus Cell ID (real time) Throughput versus # of Assigned HS-PDSCH Codes (real time) Throughput (primary, secondary, and combined)
MAC-HS Throughput versus DC-HSDPA MAC-HS Throughput (primary, secondary and combined) versus CINR (scatter plot and real-time) versus Cell ID (e.g., handover performance)
Throughput Throughput
CINR versus RSSI (scatter plot and real-time) CINR versus Modulation Scheme and/or MCS Transmit Power versus Cell ID Throughput versus Transmit Power Scheme (antenna 1 and antenna 2) October 19, 2011 | Signals Ahead, Vol. 7, Number 13 PREVIEW
UL UL
Modulation
Mother of all Network Drive Tests (LTE, DC-HSDPA, HSPA+, Rev A and Mobile WiMAX) challenges of delivering video in a mobile network modules/netbooks network performance (throughput and latency) impacts the user experience network performance benchmark results
How
Embedded TD-LTE
CoMP and LTE Advanced Green financial implications and challenges Smartphone signaling implications and LTE chipset performance benchmark test results
Going
LTE The
impact of Type 3i receivers on UE performance (includes chipset benchmark tests of leading solutions) happened to IMS? Americas (MIMO) network performance benchmark results
Whatever LTE 4G
World and GSMA MAC impact of latency Safety Options with 700MHz Rev B network performance benchmark results
HSPA+ The
Public
EV-DO LTE
chipset landscape
HTTP R HSDPA T SNR MI RSRQ R MAC-HS MOS LT WiMAX QPSK C HS-PDSC TCP dBm EV-DO R BLER 64 CINR A E-DPCH HSPA+ D MCS MC DL RSSI HS-SCC Mbps m A/B Mo 16QAM PDSCH HTTP R HSDPA T SNR MI
9,000 6,000 3,000
35:09.0
-60
35:21.0
RTT TCP dBm dB Mbps ms MOS LTE HSPA+ TD-LTE EV-DO Rev A/B Mobile WiMAX MCS IMO RI BLER 64QAM 16QAM QPSK CQI L1 DL RSCP CINR ACK NACK PDSCH HS-PDSCH HS-S Coming Soon! S E-DPCH FTP UDP HTTP RTT TCP dBm dB Mbp TE HSPA+ DC-HSDPA TD-LTE EV-DO Rev A/B M X MCS MCS SNR MIMO RI BLER 64QAM 16Q CQI L1 DL RSSI RSRQ RSCP CINR ACK NACK PD SCH HS-SCCH MAC-HS E-DPCH FTP UDP HTTP m dB Mbps ms MOS LTE HSPA+ DC-HSDPA TD Rev A/B Mobile WiMAX MCS MCS SNR MIM 4QAM 16QAM QPSK CQI L1 DL RSSI RSRQ ACK NACK PDSCH HS-PDSCH HS-SCCH MA H FTP UDP HTTP RTT TCP dBm dB Mbps ms MO DC-HSDPA TD-LTE EV-DO Rev A/B Mobile Wi CS SNR MIMO RI BLER 64QAM 16QAM QPSK C I RSRQ RSCP CINR ACK NACK PDSCH HS-PD CH MAC-HS E-DPCH FTP UDP HTTP RTT TCP dB ms MOS LTE HSPA+ DC-HSDPA TD-LTE EV-DO obile WiMAX MCS MCS SNR MIMO RIsubscription) 64Q BLER pre-order your report lIcense now (included as part of a Signals Ahead QPSK CQI L1 DL RSSI RSRQ RSCP CINR ACK N HS-PDSCH HS-SCCH MAC-HS E-DPCH FTP RTT TCP dBm dB Mbps ms MOS LTE HSPA+ TD-LTE EV-DO Rev A/B Mobile WiMAX MCS IMO RI BLER 64QAM 16QAM QPSK CQI L1 DL
5,000
70%
4,000 3,000
60% 50%
52.9%
35.7%
40% 30%
2,000 1,000 0
PHY Layer Primary Throughput (Kbps) PHY Layer Secondary Throughput (Kbps)
36:45.0 36:57.0 37:09.0 37:33.0 37:45.0 37:57.0 37:21.0 38:09.0
18.5%
35:45.0
35:33.0
35:57.0
36:09.0
36:33.0
36:21.0
-50
-40
-30
-20
lIcense costs
38:21.0
contact InformatIon
Volume 1 avaIlable now! Network and Technology Performance ($1,995) Volume 2 avaIlable now! Quantifying the User Experience ($1,495) Volume 3 Detailed Performance Analysis ($1,295, $1,495) full report all 3 volumes ($3,300, $3,995) 9
You may call us at +1 (510) 273-2439 or email us at information@signalsresearch.com and we will contact you for your billing information or respond to any further inquiries that you may have. Subscription information for our Signals Ahead research product, which includes these reports, can be found on the last page of this report. You can also visit our website at www.signalsresearch.com or write us at
Signals Research Group, LLC 10 Ormindale Court Oakland, CA 94611 October 19, 2011 | Signals Ahead, Vol. 7, Number 13 PREVIEW
17:29.0 17:37.0 17:45.0 17:53.0 18:01.0 18:09.0 18:17.0 18:25.0 18:33.0 18:41.0 18:49.0 18:57.0 19:05.0 19:13.0 19:21.0 19:29.0 19:37.0 19:45.0 19:53.0 20:01.0 20:09.0 20:17.0 20:25.0 20:33.0 20:41.0 20:49.0 20:57.0 21:05.0 21:13.0 21:20.0 21:27.0 21:35.0
In Volume 1 of a special three-part series of reports we leverage the capabilities of Accuvers complete suite of drive test tools to analyze the performance of all next-generation wireless technologies as exemplified by operator deployments in North America. In addition to looking at basic parameters, such as throughput and latency, the report looks at end-user spectral efficiency and the different network deployment philosophies of the leadng operators (e.g., their cell grid density). networks In this report we discuss the recent trends impacting the various mobile platforms that exist and what has transpired since our piece from three years ago on Web 2.0. We address the state of the mobile platforms that exist, provide our thoughts on the current and future prospects and look at the various trends that are driving the industry. provide the key takeaways from the LTE World Summit, held in Amsterdam. Spectrum fragmentation tops the list of key LTE topics, although a growing focus on the use of 1800MHz for those operators that have access to it is encouraging. VoLTE, or the lack thereof, is still on everyones minds, but in the interim CSFB isnt even working as promised. Finally, there was a lot of talk about Mobile WiMAX, but the emphasis seemed to be on how to best move away from the technology and adopt TD-LTE. collide In addition to covering the basics of heterogeneous networks (HetNet), a key LTE-Advanced (R10) feature, we present a compelling series of analytical studies which demonstrate the need for macro network offload, starting as early as 2015. We also get into the technical details of how HetNet works, including discussions on eICIC, ABS and the importance of interference cancellation in the handset. Finally, we look at what is being done with legacy 3G femtocells to limit interference-related problems that they introduce, both with the macro network and between each other. Soy Sauce In collaboration with Spirent Communications we provide results from the industrys only independent performance benchmark tests of HSPA+/HSPA chipsets. In the most recent benchmark study we tested 16 different device configurations, representing chipsets from 9 different suppliers, including new entrants, such as Samsung (HSPA+), Intel (HSPA+), MediaTek
and HiSilicon. We provide the results, based on a total of 42 HSPA+ test scenarios and 26 HSPA test scenarios.
3/15/2011 Looking beyond HSPA+: keeping up with the
Joneses Based on interviews with 3GPP member companies and a thorough review of 3GPP submissions, we offer an in-depth look at the future of HSPA+ (Release 11 and beyond). Ultimately, we conclude that many of the features that are being incorporated into LTE will find their way into HSPA+, thus blurring the performance differences between the two technologies. Latency and the impact of new features on legacy devices are two areas of prime importance where HSPA+ could face challenges relative to LTE. the Pleasure, Part II In collaboration with Accuver, who provided us with its XCAL-W drive test tool and XCAP-W post-processing software, we provide results and analysis from an extensive drive test of Telstras DC-HSDPA network. We compare DC-HSDPA with HSPA+ performance in a number of side-by-side tests. the Pleasure, Part I In collaboration with Accuver, who provided us with its XCAL-W drive test tool and XCAP-W post-processing software, we provide results and analysis from an extensive drive test of Telstras DC-HSDPA network. We compare DC-HSDPA with HSPA+ performance in a number of side-by-side tests. tion with Sanjole we examine how some of todays commercial LTE eNodeBs allocate network resources when serving multiple devices. We determine that while LTE may deliver a compelling user experience, it is largely due to an empty network and the large channel bandwidths, and that further improvements are necessary if LTE is going to support multiple users in an efficient manner.
GSMA Mobile Asia Congress We provide and discuss various data points which stem from our participation at the LTE Americas event in Dallas and the GSMA MAC event in Hong Kong. We provide an LTE market update, including TD-LTE, discuss the debate about a smart or dumb pipe strategy, and the impact of smartphones and social networking services, including the use of cloud computing, intelligent networks, network offloading and data caching.
10
12 20
4.1 downtown san francisco July 29th, 0430 - 1730 20 4.1.1 Throughput and Latency Measurements (Downtown San Francisco July 29th, 0430 1730) 21 4.1.2 HTTP Web Page Load Time Results (Downtown San Francisco July 29th, 0430 1730) 23 4.2 http web page load time results (srg headquarters deep within the oakland hills august 8, 1800-2200) 4.3 downtown oakland august 14th, 1330 - 1530
27 28 28 29 31 31
4.3.1 Throughput and Latency Measurements (Downtown Oakland August 14th, 1330-1530) 4.3.2 HTTP Web Page Load Time Results (Downtown Oakland August 14th, 1330-1530) 4.4 santa clara august 29th august 31st 4.4.1 Throughput and Latency Measurements (Santa Clara August 29th August 31st)
4.4.2 HTTP Web Page Load Time Results (Santa Clara August 29th August 31st) 32 4.5 ev-do rev a and lte 33 4.5.1 Throughput, Latency and HTTP Web Page Load Time Results EV-DO Rev A and LTE (Kansas City, June 17th, 1700-1900) 33 4.5.2 Throughput, Latency and HTTP Web Page Load Time Results EV-DO Rev A and LTE (SRG Headquarters, August 8th, 1800-2200) 34 5.0 Why Throughput Doesnt [Always] Matter 6.0 Why Latency [Almost] Always Matters 6.1 web page 101
36
40
40
6.2 tracing the real latency to various web sites by network 43 6.3 connection time latency
46 47 49 52 56 58 61 62
6.4 low latency and its Impact on the user experience 7.1 itunes and gmail
1 1
Index of figures
figure 1. Downlink Application Layer Median Throughput San Francisco (0430 1730) figure 3. Network Latency San Francisco (0430 1730)
21 21
figure 2. Downlink Application Layer Maximum Throughput San Francisco (0430 1730)
22
figure 4. Amazon Web Page Load Times by throughput + latency combinations 24 figure 5. Yahoo Web Page Load Times by throughput + latency combinations 24 figure 6. iTunes Web Page Load Times by throughput + latency combinations 26 figure 7. iTunes Web Page Load Times by operator
26 28 29 30
figure 8. CNN Web Page Load Times by throughput + latency combinations (SRG HQ) 27 figure 9. Throughput and Latency Results by Operator (Oakland) figure 10. CNN Web Page Load Times by throughput + latency combinations (Oakland)
figure 11. Amazon Web Page Load Times by throughput + latency combinations (Oakland) 30 figure 12. Yahoo Web Page Load Times by throughput + latency combinations (Oakland) figure 13. Throughput and Latency Results by operator (Santa Clara) 31 figure 14. Yahoo Web Page Load Times by throughput + latency combinations (Santa Clara) 32 figure 15. Amazon Web Page Load Times by throughput + latency combinations (Santa Clara) figure 16. Throughput and Latency Results EV-DO Rev A and LTE (Kansas City)
32
33
figure 17. Web Page Load Times by throughput + latency combinations (Kansas City) 34 figure 18. Throughput and Latency Results EV-DO Rev A and LTE (SRG Headquarters)
35 35
figure 19. Wikipedia Web Page Load Times by throughput + latency combinations (SRG Headquarters) figure 21. DC-HSDPA MAC Layer Throughput HTTP Web Browsing (San Francisco, July 29th, 0500) figure 23. HTTP Application Layer Throughput by technology (Oakland, August 14th) figure 24. LTE Application Layer Requirements an entire user experience test script figure 26. Connection Latency by technology
figure 20. iTunes Web Page Load Times by throughput + latency combinations (SRG Headquarters) 35
36
figure 22. DC-HSDPA MAC Layer Throughput HTTP Web Browsing (San Francisco, July 29th, 1700) 37
38 39
figure 25. The Theoretical Impact of Varying Throughput and Latency on Web Page Load Times 41
46 47 47
figure 27. The Impact of Ultra Low Latency and Low Throughput YouTube figure 28. The Impact of Ultra Low Latency and Low Throughput PornHub figure 29. The Impact of Ultra Low Latency and Low Throughput MSN
48 50
figure 30. Gmail and iTunes Throughput Requirements Clearwires 2x20MHz LTE Network 49 figure 31. Application Layer Throughput while using iTunes with and without a background FTP session figure 33. Gmail and iTunes Throughput Requirements AT&Ts LTE Network (Houston, September 7th) figure 35. Netflix Throughput Requirements T-Mobile DC-HSDPA Network (Oakland, August 14th) figure 32. Gmail and iTunes Throughput Requirements T-Mobiles HSPA+ Network (San Francisco, July 29th) 51
51
figure 34. YouTube and Netflix Throughput Requirements Verizon Wireless LTE Network (Oakland, August 14th)
52
53 55
figure 36. YouTube and Netflix Throughput Requirements AT&T LTE Network (Houston, September 7th) 53 figure 37. Netflix Throughput Requirements T-Mobile DC-HSDPA Network (Santa Clara, August 29th, 2100) figure 38. iTunes Throughput Requirements as a Function of Time of Day Verizon Wireless LTE Network (San Francisco, July 29th) 56
1 2
figure 39. Skype Downlink Throughput Requirements Device 1 and Device 2 on the Verizon Wireless LTE Network (Oakland, August 18th) 57 figure 40. Skype Uplink Throughput Requirements Device 1 and Device 2 on the Verizon Wireless LTE Network (Oakland, August 18th) 57 figure 41. XCAL-M Drive Test Tool in Action DL performance 58 figure 42. XCAL-M Drive Test Tool in Action UL performance
59 62 63 64 64
figure 43. iTunes Web Page Load Times by throughput + latency combinations (SRG Headquarters) 62 figure 44. iTunes Web Page Load Times by operator (SRG Headquarters) figure 45. Amazon Web Page Load Times by operator (SRG Headquarters)
figure 46. Yahoo Web Page Load Times by throughput + latency combinations (SRG Headquarters) 63 figure 47. LTE Application Layer Throughput HTTP Web Browsing (San Francisco, July 29th, 0601) figure 48. LTE Application Layer Throughput HTTP Web Browsing (San Francisco, July 29th, 1635)
figure 49. Gmail Throughput Requirements as a Function of Time of Day Verizon Wireless LTE Network (San Francisco, July 29th) 65
Index of tables
table 1. URL Analysis Size and HTTP Requests
42 43
table 2. HTTP Activity Count by URL 42 table 3. Real World Latency Calculations to Target URLs by network/technology table 4. Trace Route results by operator, technology and URL
44
13
we used the accuver Xcal-mo and Xcal-m tools to collect the underlying performance indicators and the accuver Xcap-m post-processing tool to do the analysis of the data that we collected.
14
Each operator provided us with at least two dongles, although in the case of operators, such as Clearwire, with multiple network/technology deployments (e.g., 2x20MHz LTE and 1x10MHz Mobile WiMAX), we received multiple dongles. Unlike the drive test results which we presented in Volume 1, the results shown in this report were collected from a stationary position. We took this approach in order to reduce the variability in performance that can occur when moving throughout the network. Eliminating, or at least reducing, this variability was important since the objective of the user experience tests was to determine various user experience metrics for a known set of inputs primarily throughput and latency. Testing in each market took place from as early as 4AM local time until the early evening hours. We also did a lot of user experience testing during the dead of the night when we suspect the networks were wide open. Since we were using test equipment we had the ability to determine whether or not network loading was impacting the results. Suffice it to say that in the early morning hours network loading was not a concern for any of the networks. Later in the day, network loading impacted the performance of certain networks/technologies while it was not even a consideration with other networks/technologies. We take this phenomenon into consideration when doing our analysis. Each user experience test in involving HTTP web browsing consisted of the following series of tests:
A A
60 second multi-session FTP test to determine the maximum throughput potential of the network at the time. 60 second latency test in which we pinged a local server to determine the median latency. a series of popular websites and downloading their contents with an Internet Explorer browser. In order to ensure a degree of consistency, we downloaded each web page up to 50 times and then used the median download time. The download time calculations leveraged very detailed information captured by the Accuver drive test tools. October 19, 2011 | Signals Ahead, Vol. 7, Number 13 PREVIEW
Accessing
15
60 second multi-session FTP test to determine the maximum throughput capabilities of the network at the time. The results from this test were combined with the results from the first FTP test to determine the median throughput. It was our assumption that this potential throughput was relatively constant throughout the entire test, although this assumption may not have always been correct.
like all Signals Ahead reports, we received no sponsorship or funding from the participating companies, in order to maintain our independence.
For the high-bandwidth application tests, such as iTunes, Netflix and YouTube, we conducted a multi-session FTP test to determine the maximum throughput potential of the network. This test was followed by interacting with these applications (watching a video, downloading an email attachment) and using the Accuver tools to record the underlying interactions between the mobile device and the network. Like all Signals Ahead reports, we received no sponsorship or funding from the participating companies, in order to maintain our independence. As such, we foot the bill for all of our travel expenses not to mention an inordinate amount of time and effort collecting the data and writing these series of reports. We also could not have done this report without the support of Accuver who provided us with its suite of drive test tools and post-processing software. SRG takes full responsibility for the analysis and conclusions that are documented in this report and in our forthcoming report that analyzes and compares technology- and vendor-specific performance characteristics.
michael thelander
Michael Thelander is the CEO and Founder of Signals Research Group. In his current endeavor he leads a team of industry experts providing technical and operator economics analysis for clients on a global basis. Mr. Thelander is also responsible for the consultancys Signals Ahead research product, including its widely acclaimed Chips and Salsa series of reports that focus on the wireless IC industry. Previously, Mr. Thelander was an analyst with Deutsche Bank Equity Research. Prior to joining Deutsche Bank, Mr. Thelander was a consultant with KPMG (now known as BearingPoint) and a communications officer with the United States Army. Mr. Thelander has also published numerous articles for leading trade publications and engineering journals throughout his career. He has been an invited speaker at industry conferences around the world and he is frequently quoted by major news sources and industry newsletters, including The Economist, The Wall Street Journal, Investors Business Daily, Reuters, Bloomberg News, and The China Daily. Mr. Thelander earned a Masters of Science in Solid State Physics from North Carolina State University and a Masters of Business Administration from the University of Chicago, Graduate School of Business.
16
Credit Card #
Exp Date
/ /
Mailing Address Signals Research Group, LLC ATTN: Sales 10 Ormindale Court Oakland, CA 94611 Our fax number is (510) 338-1284. Alternatively, you may contact us at (510) 273-2439 or at information@signalsresearch.com and we will contact you for your billing information. We will not process your payment until after the trial subscription period is completed. Terms and Conditions: Any copying, redistributing, or republishing of this material, including unauthorized sharing of user accounts, is strictly prohibited without the written consent of SRG.
please note disclaimer: The views expressed in this newsletter reflect those of Signals Research Group, LLC and are based on our understanding of past and current events shaping the wireless industry. This report is provided for informational purposes only and on the condition that it will not form a basis for any investment decision. The information has been obtained from sources believed to be reliable, but Signals Research Group, LLC makes no representation as to the accuracy or completeness of such information. Opinions, estimates, projections or forecasts in this report constitute the current judgment of the author(s) as of the date of this report. Signals Research Group, LLC has no obligation to update, modify or amend this report or to otherwise notify a reader thereof in the event that any matter stated herein, or any opinion, projection, forecast or estimate set forth herein, changes or subsequently becomes inaccurate. If you feel our opinions, analysis or interpretations of events are inaccurate, please fell free to contact Signals Research Group, LLC. We are always seeking a more accurate understanding of the topics that influence the wireless industry. Reference in the newsletter to a company that is publicly traded is not a recommendation to buy or sell the shares of such company. Signals Research Group, LLC and/or its affiliates/investors may hold securities positions in the companies discussed in this report and may frequently trade in such positions. Such investment activity may be inconsistent with the analysis provided in this report. Signals Research Group, LLC seeks to do business and may currently be doing business with companies discussed in this report. Readers should be aware that Signals Research Group, LLC might have a conflict of interest that could affect the objectivity of this report. Additional information and disclosures can be found at our website at www.signalsresearch.com. This report may not be reproduced, copied, distributed or published without the prior written authorization of Signals Research Group, LLC (copyright 2011, all rights reserved by Signals Research Group, LLC).
17