HUAWEI TECHNOLOGIES CO., LTD.

Bantian, Longgang District

Shenzhen518129, P. R. China
Tel:+86-755-28780808

Wireless X Labs is a brand-new platform designed to get together telecom operators, technical vendors and partners from
vertical sectors to explore future mobile application scenarios, drive business and technical innovations and build an open
ecosystem. Wireless X Labs have set up three laboratories, which aim to explore three major areas: people-to-people
connectivity, applications for vertical sectors and applications in household.

Thunder Fire is founded in 2007 by the side of West Lake in Hangzhou. After a decade’s development, Thunder Fire now
becomes a thriving team of nearly 2000 employees, combining top R&D, operation, and marketing elites. The Group has
successfully developed and released flagship games series including Ghost, Justice and Revelation, exploring its way in
action, sports and casual games. With the only game studio in China dedicated to the research and application of artificial
intelligence in the game filed, Thunder Fire has always been at the forefront of the development and application of
cutting-edge technology.

HUAWEI CLOUD

HUAWEI CLOUD, the cloud service brand of the Huawei marquee, packages the 30-plus years of expertise in ICT
infrastructure products and solutions so customers can build what they need into their profile with building block-like ease.
The brand is committed to providing stable, secure, reliable, and sustainable cloud services to help organizations of all
sizes grow in the intelligent world. Complementing the already impressive offerings, the Inclusive AI strategy from
HUAWE CLOUD allows everyone to get what they need at an affordable price point, efficiently make easy use of
cutting-edge technologies, and rest assured that their profile remains fully secure throughout the entire lifecycle.

, , are trademarks or registered trademarks of Huawei Technologies Co.,Ltd

Other Trademarks,product,service and company names mentioned are the property of thier respective owners

Copyright © 2019 HUAWEI INVESTMENT & HOLDING CO., LTD. All Rights Reserved.

GENERAL DISCLAIMER
THE INFORMATION IN THIS DOCUMENT MAY CONTAIN PREDICTIVE STATEMENT, INCLUDING BUT NOT LIMITED TO, STATEMENTS REGARDING FUTURE
FINANCIAL RESULTS, OPERATING RESULTS, FUTURE PRODUCT PORTFOLIOS, AND NEW TECHNOLOGIES. THERE ARE A NUMBER OF FACTORS THAT
COULD CAUSE ACTUAL RESULTS AND DEVELOPMENTS TO DIFFER MATERIALLY FROM THOSE EXPRESSED OR IMPLIED IN THE PREDICTIVE STATEMENTS.
THEREFORE, SUCH INFORMATION IS PROVIDED FOR REFERENCE PURPOSES ONLY, AND CONSTITUTES NEITHER AN OFFER NOR A COMMITMENT. HUAWEI
MAY CHANGE THE INFORMATION AT ANY TIME WITHOUT NOTICE, AND IS NOT RESPONSIBLE FOR ANY LIABILITIES ARISING FROM YOUR USE OF ANY OF THE
INFORMATION PROVIDED HEREIN.
CONTENTS

ASSESSMENT MODEL MODEL EQUATION

FRAMEWORK STRUCTURE MODEL APPLICATION

SUBJECTIVE EXPERIMENT TYPICAL MODEL

BACKGROUND AND ANALYSIS METHOD VALUES

02 07 10

01 04 08

Overview:
This document describes cloud gaming service experience assessment, including the requirement background, assessment model
framework, subjective experiment and analysis method, model equation structure, typical model values, and model application
scenarios. Huawei X Labs and NetEase Thunder-fire Game Business Division have established a user experience model for cloud
gaming based on the observation of player behavior and player feedbacks while playing cloud games on 5G networks. The modeling
process complies with ITU specifications and uses hierarchical mapping. This model makes up for a lack of quantitative research on
cloud gaming experience in the industry. The cloud gaming experience model provides reference and guidance for 5G network
construction and industry development planning. In addition, cloud gaming content suppliers, network operators, and hardware
manufacturers can adjust and optimize service and product strategies based on user experience. It is worth mentioning that this
assessment model algorithm will be released and shared in the form of free software development kits (SDKs). Partners in the industry
are welcome to try and promote its development.
Cloud Gaming Experience Model (Cloud gMOS) Cloud X Service Experience Model Series

ASSESSMENT MODEL
BACKGROUND
FRAMEWORK

Cloud gaming is a game model based on cloud computing. In this model, game picture are calculated and rendered on Cloud gaming shares three aspects in common with cloud VR when it comes to user experience: audio-visual quality,
the cloud. After compression, rendered picture are transmitted to clients through the network. Therefore, high-end interaction quality, and consistency/integrity of audio-visual experience.
processors, graphics cards, and GPUs are no longer needed; clients only need to provide certain streaming media
Figure 2-1 shows the framework of the experience assessment model for cloud gaming. Based on hierarchical mapping,
decoding and playing capabilities. This greatly reduces the threshold for users to obtain high-quality games.
the framework can be divided into the influencing factor input layer, sub-perception experience layer, and comprehensive
Similarly, 5G networks, in terms of accessibility and local-like operations, will substantially improve user experience of experience layer.
cloud gaming. Cloud gaming services are one of the important businesses of Cloud X. The data released by market
Figure 2-1 Framework of the experience assessment model for cloud gaming
survey firm, IHS Markit, indicates that the cloud gaming industry has considerable potential for development.
Influencing factor layer Sub-experience layer Comprehensive
Figure 1-1 Estimated global cloud gaming market scale (input layer) experience layer

Cloud gaming ($ million) Source:IHS Markit l.11 Picture bit rate

3000

l.12 Picture frame rate

2500 Picture resolution

l.13
O.21
l.14 Screen refresh rate Picture quality
2000

l.15 Screen resolution

O.31
1500 l.16 Picture codec Audio-visual
quality

1000 l.18 Audio bit rate

O.22
Number of
l.19 Audio quality
audio channels
500 O.41
l.110 Audio codec Comprehensive
experience
Asynchronous
0 l.111
audio and video
2018 2019 2020 2021 2022 2023

Research on cloud gaming experience mainly used to focus on the preliminary qualitative analysis of picture l.112
Average
stalling duration
quality, consistency/integrity, while lacked systematic and in-depth quantitative research on interaction quality and
O.32
comprehensive experience. Stalling Consistency/
l.113
frequency Integrity
To build a complete cloud gaming experience model, Huawei X Labs collaborated with NetEase Thunder-fire Game
Business Division to conduct an in-depth analysis. They invited test players to use cloud computers to play games such as l.114 Packet loss rate

Justice and Meteor Butterfly Sword on 5G networks. They implemented subjective experiments based on the observation of
O.33
players' behavior and player feedbacks while complying with ITU specifications. With the support of subjective experimental Operation Interaction
l.115
data, a unified user experience assessment model for cloud gaming is trained and established, which provides reference response Delay quality

and guidance for 5G network construction and industry development from the perspective of user experience.

01 02
Cloud Gaming Experience Model (Cloud gMOS) Cloud X Service Experience Model Series

Table 2-1 lists input parameters and acronyms or abbreviations of the assessment model.

Table 2-1 Assessment model input parameters SUBJECTIVE EXPERIMENT

ID Parameter Description
Acronym or
Abbreviation AND ANALYSIS METHOD
I.11 Picture bit rate Average picture bit rate (bit/s) Br

I.12 Picture frame rate Number of picture frames per second (fps) FR
The subjective experiment of cloud gaming user experience is also based on ITU-T P.913, using the single-stimulus
I.13 Picture resolution Number of horizontal and vertical pixels in an image Rh,Rv
method and absolute category rating (ACR).

I.14 Screen refresh rate Screen refresh times per second RR Figure 3-1 shows the subjective experiment platform of cloud gaming, including cloud computer, network impairment
emulator, client, and display devices. The subjective experiment platform provides game content with different
I.15 Screen resolution Number of horizontal pixels in monocular screen RSh resolutions, frame rates, and picture quality. Under the control of the network impairment emulator, the platform provides
test scenarios with different picture integrity and operation response delay.
I.16 Picture codec H.265/HEVC, H.264/AVC, VP9 Visual Codec
Figure 3-1 Architecture of the subjective experiment platform
I.18 Audio bit rate Average audio bit rate (Kbit/s) ABr

I.19 Number of audio channels Stereo sound (2), spatial sound (8) SP

I.110 Audio codec Audio encoding format (for example, AAC) Audio Codec Motion data

I.111 Asynchronous audio and video Delay of asynchronous audio and video (second) tasyn
Impairment
Total stalling duration during a single playback, including the
I.112 Average stalling duration
initial buffering duration (second)
Tr emulator
Stalling frequency during a single playback (number of stalling
I.113 Stalling frequency RF
times/playback duration)
Audio and video data
I.114 Packet loss rate Data loss rate at the application layer ppl

Delay between the user operation instruction and operation

I.115 Operation response Delay tod
response (ms)
Client Cloud computer

Table 2-2 lists the parameters and acronyms or abbreviations of the assessment model. The details are as follows:

Table 2-2 Assessment model output 1.The impact of different display devices on user experience is measured by PPD. A corresponding assessment model is
then established accordingly.
Acronym or
ID Description Value Full Name The game picture can be shown on the screen of a monitor or of a mobile phone. With the optimal viewing distance, the
Abbreviation
resolution that can be perceived by the test user is converted into PPD. Generally, for example, the optimal viewing
O.21 Picture quality 1–5 points MOSV Visual
distance of a display screen no smaller than 10 inches should be twice the height of the display screen; while to a display
screen less than 10 inches, for example, a mobile phone, the optimal viewing distance should be four to six times the
O.22 Audio quality 1–5 points MOSA Audio
width of its screen. (In this subjective experiment, four times is required.)
O.31 Audio-visual quality 1–5 points MOSAV Audio-visual Taking a 32-inch 4K monitor as an example, the equivalent PPD can be calculated based on Figure 3-2 and the
following equation:
O.32 Consistency/Integrity 1–5 points MOSRP Re-buffering/Packet Loss
PPD = Ph/(2*arctan(W/2/D)*180/Pi)=80.47
O.33 Interaction quality 1–5 points MOSI Interaction
Ph is the number of horizontal pixels, W is the width of the display screen, and D is the optimal viewing distance (twice
O.41 Comprehensive experience 1–5 points MOSC Comprehensive
the display height).

03 04
Cloud Gaming Experience Model (Cloud gMOS) Cloud X Service Experience Model Series

Figure 3-2 32-inch 4K monitor PPD conversion 2.Figure 3-3 shows the basic operation response delay. Extra hysteresis can be set through the network impairment
emulator to simulate the impact of different response delays on user experience.

Figure 3-3 Operation response delay

W: 27.90 inches
Client Server (cloud computer)
1 2
Resolution: 3840*2160 User command
Collection/
Motion input Input processing
Transmission
32 H: 15.68 inches
inc
he
s
Logical computing 3

Graphic rendering
9 8

Display on
Client rendering
the monitor Graphic capturing 4
Downlink transmission
D: 31.36 inches of coded graphics
Graphic decoding Graphic coding 5

7 6

The operation response delay includes:

Table 3-1 lists PPD of a 32-inch monitor and a Huawei Mate20 mobile phone screen with different resolutions. 1.instruction data collection delay; 2.uplink transmission delay of instruction data;
3.instruction receiving, logical operation, and graphic rendering delay; 4.graphic capturing delay;
Table 3-1 Equivalent PPD comparison 5.graphic coding delay; 6.downlink transmission delay of coded graphics; 7.graphic decoding delay;
8.client rendering delay, including the delay in waiting for the video buffer after decoding, color processing, and so on
Display Height Width Diagonal Optimal Viewing Picture
PPD layer overlay; 9.display delay of a monitor.
Device (Inch) (Inch) (Inch) Distance (Inch) Resolution

854*480 17.62

1280*720 26.41

32-inch 27.90 15.68 32 31 1920*1080 39.62

monitor

2560*1440 53.37

3840*2160 80.47

854*480 29.33

Mate20 5.88 2.83 6.53 11.32 1280*720 43.96

1920*1080 65.94

05 06
Cloud Gaming Experience Model (Cloud gMOS) Cloud X Service Experience Model Series

MODEL EQUATION STRUCTURE TYPICAL MODEL VALUES

The following describes the equation of the comprehensive experience assessment model of cloud gaming. The 1.Typical PPD effect on picture quality MOSV
equation is consistent with that in the VR experience assessment model. The correlation coefficient is fitted based on the Based on H.264 encoding, when BPP is 0.133 and the frame rate is 120 fps, the typical PPD effect on the picture quality MOSV
actual test result. is shown in Figure 5-1. As shown in the figure, when PPD is 20, the picture quality MOSV is about 3.57 points; when PPD is 60,
the picture quality MOSV is about 4.73 points; and when PPD is over 60, the picture quality MOSV slightly increases.
(
MOS C = min max ( (MOS AV-1) ·( 1-v 28(5-MOS I)-v 29(5-MOS RP) ) +1,1 ) ,5 ) 1
MOSV Figure 5-1 PPD effect on picture quality MOSV
In the equation, MOSC is a comprehensive experience of cloud gaming, MOSAV is audio-visual quality, MOSI is interaction 5.00
quality, MOSRP is audio-visual consistency/integrity, and v28, and v29 are model coefficients. The comprehensive experience
4.50
is based on the audio-visual quality MOSAV, the interaction quality MOSI, and consistency/integrity MOSRP are
4.00
sub-perception items of experience impairment.
3.50
MOS V = f 1( Visual Codec,Br,FR,(R h,R v) ) 2
3.00

MOS A = f 2(Audio Codec,ABr,SP) 3 2.50

2.00
MOS AV = f 3(MOS V,MOS A,t asvn) 4
1.50

MOS I = f 4(t od) 5 1.00

PPD
0 20 40 60 80 100 120

MOS RP = f 5(T r,RF,ppl) 6

2.Typical resolution effect on picture quality MOSV under the optimal viewing distance
Based on H.264 encoding, when BPP is 0.133 and the frame rate is 120 fps, images with different resolutions can be
viewed on a monitor. And the typical resolution effect on picture quality MOSV is shown in Figure 5-2. In the case of high
frame rate, 1080P game picture can correspond to a higher PPD, and the MOSV is 4.5 points.

MOSV Figure 5-2 Effect of picture resolution on MOSV

5.00

4.50

4.00

3.50

Game Picture Resolution

3.00

2.50

2.00

1.50

1.00
480P 720P 1080P 2K 4K

07 08
Cloud Gaming Experience Model (Cloud gMOS) Cloud X Service Experience Model Series

3.Typical effect of frame rate on picture quality MOSV

Based on H.264 encoding, when BPP is 0.133 and PPD is 120, typical effect of the frame rate on picture quality MOSV is
shown in Figure 5-3. When the frame rate is 30 fps, picture quality MOSV is about 3.35 points; when the frame rate is 60
fps, picture quality MOSV is about 4.41 points; and when the frame rate is 90 fps, picture quality MOSV exceeds 4.80 MODEL APPLICATION
points. When the frame rate exceeds 90 fps, picture quality MOSV increases slowly.

MOSV Figure 5-3 Effect of frame rate on picture quality MOSV

5.00
4.50
4.00 Based on the following conditions, Table 6-1 describes the comparison of the experience of cloud gaming with different
3.50 display devices and network conditions.
3.00 1.Based on H.264 encoding, BPP is 0.21.
2.50 2.If the display screen is no smaller than 10 inches (large screen), such as a monitor, the optimal viewing distance should
2.00 be twice the height of the display screen. If the display screen is no larger than 10 inches (small screen), such as a
mobile phone, the optimal viewing distance should be four times the width of the display screen. (If a game is viewed on
1.50
FPS a display screen with an appropriate height and width from the optimal viewing distance, despite the size of the display
1.00
0 20 40 60 80 100 120 screen, the equivalent PPD can be obtained so long as the monitor has the same resolution.)
3.Data transmission has no impact on audio-visual integrity.
4.Typical effect of the operation response delay on the interaction quality MOSI
4.Audio encoding algorithm: AAC-LC; audio fixed bit rate: 140 Kbit/s.
When the operation response delay is less than 50 ms, the interaction quality MOSI is 5 points; when the operation
Table 6-1 Comparison of cloud gaming user experience under different parameter conditions
response delay is 100 ms, the interaction quality MOSI is 4.03 points. When the operation response delay is 200 ms, the
interaction quality MOSI is 2.92 points. Game Picture Average Operation
Display Equivalent Frame
Picture Bit Rate MOSV MOSAV Response MOSI MOSC
MOSI Figure 5-4 Typical effect of the operation response delay on the interaction quality MOSI Device PPD
Resolution Rate (fps) (Mbit/s) Delay (ms)
5.00

Operation Response Delay(ms)

4.50 50 5.00 2.95
4.00 30 7 2.84 2.95 70 4.62 2.77
3.50 100 4.03 2.48
1280*720 26.41
3.00 50 5.00 3.75
2.50 60 13 3.74 3.75 70 4.62 3.49
2.00 100 4.03 3.08
1.50 50 5.00 3.23
1.00 30 15 3.16 3.23 70 4.62 3.02
0.00 100.00 200.00 300.00 400.00 500.00 600.00 100 4.03 2.69
Display 1920*1080 39.62
5.Typical effect of packet loss rate on integrity MOSP 50 5.00 4.12
screen
Different data transmission models have different relationship models of packet loss rate and image integrity. Based on 60 30 4.16 4.12 70 4.62 3.82
no
TCP+RTP transmission model, typical effect of TCP packet loss rate on integrity MOSP is shown in Figure 5-5. When the 100 4.03 3.36
smaller
TCP packet loss rate is 0.1%, the audio-visual integrity MOSP is 5 points; when the packet loss rate is 0.3%, the audio-visual than 10 50 5.00 3.36
integrity MOSP is 3.68 points; and when the packet loss rate is 1%, the audio-visual integrity MOSP is 1.14 points. inches 30 27 3.31 3.36 70 4.62 3.14
Figure 5-5 Effect of packet loss rate on integrity MOSP (large
MOSP 100 4.03 2.79
5.00 screen) 2560*1440 53.37
50 5.00 4.29
4.50
60 53 4.36 4.29 70 4.62 3.98
4.00
100 4.03 3.49
3.50
Packet Loss Rate(%)

50 5.00 3.47
3.00
30 60 3.43 3.47 70 4.62 3.24
2.50
100 4.03 2.87
2.00 3840*2160 80.47
50 5.00 4.43
1.50
60 120 4.52 4.43 70 4.62 4.11
1.00
0.0 0.2 0.4 0.6 0.8 1.0 1.2 100 4.03 3.60

09 10
Cloud Gaming Experience Model (Cloud gMOS) Cloud X Service Experience Model Series

50 5.00 3.04
30 3 2.94 3.04 70 4.62 2.84

854*480 29.33
100
50
4.03
5.00
2.54
3.86 CONCLUSION
60 6 3.87 3.86 70 4.62 3.59
100 4.03 3.16
50 5.00 3.29
30 7 3.22 3.29 70 4.62 3.07
100 4.03 2.73
Display
1280*720 43.96
50 5.00 4.19
In the 5G era, the cloud gaming industry is destined to make tremendous progress. With
screen 60 13 4.24 4.19 70 4.62 3.88 the experience assessment model, cloud gaming content suppliers, network operators,
no larger 100 4.03 3.41
than 10
and hardware manufacturers can adjust and optimize service and product strategies
50 5.00 3.43
inches
30 15 3.38 3.43 70 4.62 3.20
based on user experience to effectively improve competitiveness. Finally, to strengthen
(small
screen) 100 4.03 2.84 industry cooperation, Huawei X Labs will release and share the assessment model
1920*1080 65.94
50 5.00 4.38 algorithm in the form of free SDK. Friends and partners in the cloud gaming industry are
60 30 4.45 4.38 70 4.62 4.05
100 4.03 3.55
welcome to try and feedback suggestions.
50 5.00 3.49
30 27 3.45 3.49 70 4.62 3.25
Method of contact: XLabs@huawei.com
100 4.03 2.88
2560*1440 87.92 To learn more about model details and have an experience, scan the following QR code to access the introduction web
50 5.00 4.46
60 53 4.54 4.46 70 4.62 4.12 page of the Cloud X service experience assessment model.
100 4.03 3.61
According to the table, we can observe the development of the cloud gaming industry and draw a blueprint for the
development of this industry in the future.
1.720P or 1080P, with the frame rate of 30 fps, is the resolution of games that can be provided by mainstream cloud
gaming platforms. Whether the game is displayed on a mobile phone screen or a monitor, the comprehensive experience
MOSC is between 3.2 and 3.5 points, and is at a level that has just exceeded the acceptable one.
2.For 5G cloud gaming, it is recommended that an experience of 1080P@60 fps be given first, corresponding to
professional players using local computers. In most scenarios (operation response delay 70 ms), it is expected that a
4-point experience can be realized on mobile phones.
3.For a large display screen, when the resolution of a game picture is 2K (2560x1440), the frame rate is 60 fps, and the
operation response delay is 50 ms, the comprehensive experience MOSC can reach 4.29 points, and user experience is
good, which can be used as a goal in the next two years.
4.When the resolution of a game image is 4K (3840x2160), the frame rate is 60 fps, and the operation response delay is
50 ms, the comprehensive experience MOSC is close to 4.5 points, which can be seen as a long-term goal.
Figure 6-1 shows the status quo of cloud gaming experience and industry development.
Figure 6-1 Status quo of cloud gaming experience and industry development
Large screen 2K @60 fps
This document is co-written by Huawei X Labs , NetEase Fire Game Division and Huawei Cloud.
Operation response delay 50 ms
Small screen 720P @30 fps
Due to the rapid development of related technologies in the 5G E2E industry, this document is only for reference and cannot be used as a basis for investment
Large screen 1080P @30 fps Small screen 1080P @60 fps Large screen 4K @60 fps
Operation response delay 70 ms research or decision-making. All statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.
Operation response
Small screen 1080P @30 fps We may supplement, correct and revise relevant information without notice, but does not guarantee immediate release of the revised version. All statements,
delay 50 ms
information, and recommendations in this document do not assume any responsibility for any direct or indirect investment profit and loss.

3 3.5
Status quo of experience
This document is an intellectual property of Huawei. No part of this document may be reproduced, transmitted or released in any form or by any means without
4 4.5 Cloud gMOS prior written consent. If any content of this report is released by any other party in the form of reference, Huawei should be attributed to as the source. Any
citation, deletion and modification shall not violate the original meaning of this report.
Longterm goal
Initial stage of 5G Goal in the next 2 years For any question or suggestion, please contact: XLabs@huawei.com.

11 12