Professional Documents
Culture Documents
AWTG Benchmarking
AWTG Benchmarking
Executive Summary
January 2013
www.awtg.co.uk
Content
OverviewandBackground
Equipment
Methodology
SampleResults
OverviewandBackground
AWTGhascarriedapilotmobileservicebenchmarking.Thebenchmarkingaimsatmultidimensionalandmultilevel
analysis
ScopeofWork
BenchmarkingfourmajoroperatorsintheUK
InabusyChristmasperiodandbusyareaofapproximately0.6km2 inLondonthroughmorethan12kmwalktest
Fullysynchronizedtestingusingstateofthearttestequipment anddedicatedservers
Forthefollowingscenarios
3Gvoiceandvariousdataservices
RecentlylaunchedLTE(4G)service
OutdoorandIndoortesting
ServicesonnewBorisbusandtraditionalLondonDoubleDeckerbus
PublicWiFi performanceanalysesagainstLTEnetwork
Page3
HardwareandSoftwareTools
AWTG used SwissQual benchmarking tool called Ranger for this project and the related post processing tools also
AWTG QoE and Broadband tester has been used to capture detailed log files especially for WiFi benchmarking
furthermore AWTGs spectrum interference analyzer tool has been used for the more detail analyses.
Qualityof
Experience
SwissQual
DiversityRanger
(SwissQual)
Qualityof
Experience
PostProcessing
4xGalaxyS2
2xO2SIM
2xTMobileSIM
2xThreeSIM
2xVodafoneSIM
4xHuaweiDonglesE398
SwissQual NQDI
MapInfo
WireShark
CascadePilot
(AWTG Tool)
Page4
Methodology
Threemaintestcasescoveredonthisprojectwhichincludeindoor,outdoorandbustests
Multistoryinbuildingtesting
Page5
3GRadioSummaryReport(OutdoorWalkTest)
HeatMaps&Statistics
OP1
OP2
OP3
OP4
AverageRSCP
OP1
OP2
OP3
OP4
55
dBm
60
61.98
65
70
75
68.16
Min=127.26
Max= 21.45
SD=37.33
OP2hasthebestRSCPinthetestarea
68.78
Min=107.56
Max= 15.82
SD=26.04
Min=105.55
Max= 38.21
SD=26.57
70.12
Min=102.75
Max= 26.38
SD=38.92
Page6
3GRadioSummaryReport(OutdoorWalkTest)
HeatMaps&Statistics
OP1
OP2
OP3
OP4
AverageEc/Io
dB
OP1
0
2
4
6
8
10
12
8.04
Min=26.73
Max= 0.88
SD=9.32
OP3hasthebestEc/IoandthelowestSD
OP2
8.99
Min=26.76
Max= 1.39
SD=8.22
OP3
OP4
7.91
Min=31.192
Max= 0.112
SD=7.37
9.81
Min=26.8
Max= 0.51
SD=11.04
Page7
FTP(OutdoorWalkTest)
HeatMaps&Statistics
OP4FailedFTPUploadTest
AverageUplinkThroughput
Kbps
Min=11.1
Max= 1496.8
800
700
600
500
400
300
200
100
0
735
Min=11.1
Max= 2337 Min=11.5
Max= 2526
545
482
NA
OP1
OP2
OP3
OP4
FTPUploadTestSuccessRate
3000
2500
OP4FTPUplinkTest
Max=2526.3 Min=11.5Avg=482
SuccessfulTest=58FailedTest=47
73%
80%
57%
60%
55%
40%
Kbps
2000
1500
20%
1000
500
NA
0%
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
LowsuccessrateandthroughputvalueforFTPUL
OP1
OP2
OP3
OP4
Page8
FTP(OutdoorWalkTest)
HeatMap&Statistics
OP4FTPDownlinkTest
Outdoor WalkTestFTPDownlinkSuccess
Rate
99%
100%
99%
OP2
OP3
OP4
100%
80%
60%
40%
29%
20%
0%
OP1
FTPDownlinkAverageThroughput
Min=235.3
Max= 6106.4 MiMin= 70.7
2,608 Max= 11502.3
4000
Kbps
3000
2000
1000
Min=179.2
Max= 1294.8
Min=31.9
Max= 10834.4
3,663
1,925
737
0
OP1
AcceptablesuccessrateandthroughputvalueforFTPDL
OP2
OP3
OP4
Page9
3GRadioSummaryReport(OutdoorWalkTest)
CallStatistics
100%
100%
98%
96%
94%
92%
90%
88%
86%
84%
Dropped
CallStatistic CompletedCallRate%
99%
96%
Failed
CallStatistic Dropped&FailedCall%
7.3%
8%
6%
90%
3.8%
4%
2.4%
1.4%
2%
0%
0%
0%
0%
0%
OP1
OP2
OP3
OP1
OP4
AccessibilityVoice
100%
100%
99%
98%
97%
96%
95%
94%
93%
OP2
OP3
OP4
Retainability Voice
100%
100%
100%
100%
OP3
OP4
100%
99%
97%
97.8%
98%
96%
97%
96%
95%
OP1
OP2
OP3
OP4
OP1
OP2
OverallradioperformanceforOP2islowerincomparisonwithother
operators
Page10
VoiceMOS(WalkTest)
HeatMap&Statistics
OP5MOSHeatMap
AverageMOS
5
4
3.37
3.23
3.41
OP2
OP3
3.45
2.82
3
2
1
0
OP1
Sec
OP5
AverageSetupTimeforcallscompleted
8
6
OP4
5.28
6.41
5.80
5.35
3.53
4
2
0
OP1
OP2
OP3
TheoverallMOSvalueforOP5ishigherthanotheroperators
howevercallsetuptimeishigher
OP4
OP5
Page11
HandoverStatistics
HeatMap&Statistics
100%
100%
98%
100%
100%
100%
100%
99.96%
OP1
OP2
OP3
OP4
100%
95%
95%
90%
90%
85%
3GOP4Handover
IntegritySuccessful3G
Handover
IntegritySuccessful2G
Handover
85%
NA
80%
80%
OP1
UMTS2100
OP2
GSM1800
OP3
OP4
UMTS900
GSM900
ServingTechnology
100%
100%
80%
60%
51.6% 48.4%
40%
20%
48.5%
47%
33.1%
27.5%
21.3%
0% 0%
4.2%
0% 0% 0%
9.6% 8.9%
0%
OP1
OP2
OP3
OP4
3GOP4istheonlyoperatorthathassomefailedhandovers
Page12
IndoorTest
Statistics
OP1
AverageThroughput(Kbps)
3GOP4MOS
OP2
OP3
OP4
WebBrowsingAverageThroughput
3000
2500
2,102
2,048
2000
1,635
1,377
1500
1,267
999
882
1000
411.6
267
500
342
732
123.2
1027.2
813.6
264
218
OP1
OP2
OP3
100%
100%
80%
OP4
100% 100%
79%
AverageSuccessRate
100%
94%
93%
100%
87%
76%
73%
100%
93%
93%
71%
60%
40%
27%
20%
0%
www.bbc.co.uk
www.google.com
www.msn.com
TheoverallwebexperienceforOP4usersislowercomparedwith
otheroperators
www.yahoo.com
Page13
BusTest
HeatMaps
3GOP4MOSNormalBus
3GOP4MOSBorisBus
3GOP4AverageRSCP
Walk
NormalBus
BorisBus
68
72
74
76
Sec
db
70
70.5
3GOP4MOSWalk
BorisBusAverageSetupTimeforcallscompleted
4.61
6.06
5.32
73.8
3.34
3.29
3.43
3.34
OP1
OP2
OP3
OP4
2
73.7
4
3
3.82
BorisBusAverageMOS
0
OP1
OP2
OP3
OP4
NosignificantdegradationbetweenNormalbusandBorisbus
Page14
LTEOPRadioReport
HeatMap&Statistics
LTEOPRSRP
RSRP Distribution
Ave =-91.7dbm Max= -65.9dbm Min=-121.4dbm
PDF
CDF
10%
120%
9%
100%
8%
7%
80%
5%
60%
4%
40%
3%
2%
20%
1%
0%
-46--44
-50--48
-54--52
-58--56
-62--60
-66--64
-70--68
-74--72
-78--76
-82--80
-86--84
-90--88
-94--92
-98--96
-102--100
-106--104
-110--108
-114--112
-118--116
3G
-122--120
-126--124
LTE
-130--128
0%
[dbm]
TherewasgoodLTEcoverageinmostofthecoveredarea
Page15
CDF
6%
LTEOPRadioReport
HeatMap&Statistics
LTEOPRSRQ
RSRQ distribution
Ave =-5.7dbm Max= -2dbm Min=-15.3dbm
5%
20%
0%
0%
3G
[db]
TherewasgoodLTERSRQinmostofthecoveredarea
Page16
CDF
40%
-4--3
10%
-6--5
60%
-8--7
15%
-10--9
80%
-12--11
20%
-14--13
100%
-16--15
25%
-18--17
120%
-20--19
LTE
CDF
30%
-22--21
LTE&3GOPsFTPDL/UL(WalkTest)
Statistics
FTPDownlinkAverageThroughput
FTPDownlinkSuccessRate
Min=1199.7
Max= 24648.1
9,423
10000
Kbps
Min=235.3
Max= 6106.4
6000
4000
2,608
Min=70.7
Max= 11502.3
Min=31.9
Max= 10834.4
100%
3GOP2
3GOP3
3GOP4
LTE
40%
1,925
20%
NA
3GOP1
0%
3GOP2
3GOP3
3GOP4
FTPUplinkAverageThroughput
8,825
NA
3GOP1
LTE
Min=2438.1
Max= 19030.7
10000
FTPUplinkSuccessRate
73%
57%
60%
6000
Min=11.1
Max= 1496.8
4000
2000
735
NA
3GOP1
Min=11.1
Max= 2337
545
Min=11.5
Max= 2526
3GOP3
20%
482
3GOP4
55%
40%
0%
3GOP2
100%
100%
80%
8000
Kbps
99%
60%
3,663
2000
100%
80%
8000
99%
100%
LTE
NA
3GOP1
3GOP2
3GOP3
3GOP4
FTPDL/ULThroughputforLTEOPismuchhigherthanall3GOPs
LTE
Page17
LTEOPRadioReport
HeatMaps
LTEOPRSRP
LTEOPPingTest
LTEOPFTPDL
LTE
3G
FailedpingtestsandlowerFTPthroughputmayhavebeencaused
bypoorcoverage
Page18
LTEOPand3GOP4RadioReport
HeatMaps
LTEOPRSRP
LTE
3GOP4RSCP
3G
3GOP4hasbettercoveragecomparedtoLTEOP
Page19
LTEOP&3GOP4FTPDownlink(WalkTest)
HeatMaps
LTEOPFTPDownlink
3GOP4FTPDownlink
FTPDownlinkThroughputforLTEOPismuchhigherthan3GOPs
Page20
LTEOP&3GOP4FTPUplink(WalkTest)
HeatMaps
LTEOPFTPUplink
3GOP4FTPUplink
FTPUplinkforLTEOPismuchhigherthan3GOPs
Page21
LTEOP&3GOP4WebThroughput(WalkTest)
HeatMaps
LTEOPWebThroughput
3GOP4WebThroughput
HigherwebthroughputforLTEOPcomparedto3GOPs
Page22
LTE&3GWebThroughputUsingHandset(WalkTest)
Statistics
LTEOP1
3GOP2
3GOP3
100
3GOP4
7.2
8.0
68.8
6.0
40
NA
7.7
15.1
NA
17.5
9.4 11.2
13.0
5.0
4.5
4.0
2.0
3.1
0.4
4.1 4.2
1.8
0.9
0.3
0.0
BBC
MSN
Yahoo
98.8%
100%
95.3%
93.7%
BBC
100%
99.6%
99.7%
92.3%
92%
90%
80%
LTEOP1
3GOP2
3GOP3
3GOP4
88%
LTEOP1
3GOP2
3GOP3
3GOP4
ms
94%
85%
Min=90
Max= 4943
SD=462
98.9%
96%
87.2%
Yahoo
AverageDelay
98%
90%
MSN
DelayTestSuccessRate
WebBrowsingSuccessRate
95%
3GOP3
TimetoLoad
60
20
3GOP2
WebThroughput
83.1
Sec
Kbps
80
LTEOP1
3GOP4
600
500
400
300
200
100
0
Min=53
Max= 4973
SD=561
480
Min=55
Max= 4918
SD=552
462
430
3GOP3
3GOP4
Min=40
Max= 877
SD=59
75
LTEOP1
3GOP2
WebthroughputforLTEOPismuchhigherthanall3GOps,alsoLTE
OPhasthelowestdelay
Page23
LTEandWiFiComparisonTest(AWTGBroadbandTester)
StaticTest
LTE
TestLocations
LTE
O2WiFi
OutdoorWiFi
FTPUplinkThroughput
12
10.9
FTPDownlinkThroughput
8
5.3
4.0
Location1
50
40
22.5
19.1
13.7
ms
Mbps
20
47.36
O2WiFi
OutdoorWiFi
26.87%
30%
43.75
32.29
Location2
PacketLoss%
23.87%
36.84
20%
16.12%
11.40%
20
10
Location2
Location1
LTE
30
10
Location1
Delay
36.2
30
Location2
OutdoorWiFi
O2WiFi
InternetSpeedDownload
40
5.4
LTE
6.7
6.1
OutdoorWiFi
O2WiFi
Mbps
10
Mbps
10
LTE
11.0
12
10.0
OutdoorWiFi
O2WiFi
10%
Location1
Location2
0%
Location1
OverallFTPDownlinkperformancewithLTEislowerthanWiFi whilethe
UploadPerformanceismuchhigher
Location2
Page24
LTEandWiFiComparisonTest(AWTGBroadbandTester)
StaticTestYouTubeSessionErrorAnalysis
LTE
TestLocations
O2WiFi
OutdoorWiFi
AverageNumberofErrors
189
200
150
100
172
Location1O2WiFiError
OutdoorWiFi
OutdoorWiFi
Location2O2WiFiError
5%
5%
7%
7%
125
71
13%
16%
50
0
70
1.2
64
63.4
60
1.02
Location1LTEError
Location2LTEError
1
45.18
45.8
Mbps
Sec
50
Location2
YouTubeThroughput
TimetoLoad
73%
75%
Location1
40
30
20
0.8
9%
0.4
10
0
Location2
16%
0.72
0.6
0.2
Location1
0.73
31%
44%
54%
20%
22%
Location1
4%
Location2
Page25
LTEandWiFiComparisonTest(AWTGBroadbandTester)
StaticTest
LTE
O2WiFi
OutdoorWiFi
LTE
Location1TimetoLoad
10
9.0
5.2
Mbps
Sec
5.3
5.2
4.8
Location1WebThrougput
7.3
OutdoorWiFi
O2WiFi
3.17
2.35
1.73
1.14
MSN
Yahoo
BBC
Location2TimetoLoad
6.4
4.5
7.6
6.9
4.3
Mbps
MSN
Yahoo
Location2WebThroughput
9.2
10
Sec
1.89
0
BBC
1.93
1.93
2.13
2.40
2.33
1.60
1.24
2
0
0
BBC
MSN
Yahoo
BBC
MSN
OverallperformanceofLTEandWiFi technologiesarecomparablein
bothlocationswhiletimetoloadforLTEisbetterthanWiFi
Yahoo
Page26
Thank You
Registered in UK No. 5793772HQ in London UK
Office: +44 (0) 208 799 0368
Fax: +44 (0) 208 728 9610
enquiries@awtg.co.uk