Journal of Physics: Conference Series

PAPER • OPEN ACCESS

Cloud gaming controls

To cite this article: Andrei Kazakov 2021 J. Phys.: Conf. Ser. 2134 012018

View the article online for updates and enhancements.

This content was downloaded from IP address 150.129.92.22 on 20/12/2021 at 15:49

ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

Cloud gaming controls

Andrei Kazakov
420500, Russia, Republic of Tatarstan, Innopolis, Universitetskaya str., 7 bld., LP
Technologies - Loudplay
E-mail: andrey.kazakov@loudplay.io

Abstract.
With an increasing interest in cloud gaming, streaming services requires different techniques
for preprocessing of data coming from users. Such techniques should overcome the problem of
delays while transferring data through the network. In this work, we are focusing on sending
the correct states of users’ devices (keyboard, mouse, and gamepad) to avoid sticking keys while
playing games through our streaming software. We present a simple method that can solve the
problem of sticky keys. The evaluation of the proposed approach shows that using this method
we can increase the number of successfully registered keys in a server.

1. Introduction
Nowadays steaming software is becoming more popular [10]. Due to fast development of
information technologies, it has become used in many areas of our lives: education [9],
entertainment [8], medicine [14], and many others. Only two biggest platforms as Twitch and
YouTube gather about 9 million active streamers and 2 billion users, respectively [3, 4]. The
increased interest in gaming content on these platforms is indicative of user interest in video
games. However, not all users can play these games due to the high hardware requirements. To
overcome this obstacle, it was suggested to play games remotely. For example, steam remote
play [5] offers the ability to play games using a mobile phone. But, in this case, the user still
needs to have a powerful personal computer (PC) to run the game on.
Many remote gaming companies, like Loudplay [2], provide services for users to run
demanding games right on powerful virtual machines. These machines could have various
configurations such as more CPU, RAM, or video memory. Such an approach allows playing
games with higher settings, on higher resolutions, on outdated devices without the necessity to
buy an expensive computer. Loudplay provides access to a virtual PC - environment (usually
Windows OS) where users can install any software they need: games or another resource-
consuming software such as a graphic engine.
Nevertheless, even in such advanced technology as cloud gaming, there still exist limitations.
Users often complain about sticky keys while playing. It usually happens due to the usage of
UDP protocol [15] for transferring control states - information of pressed buttons, the position of
the mouse, or plugged gamepads. UDP does not guarantee the ordering or delivery of messages.
When the user clicks a button, the network receives information about 2 events: press and release
states. When release message delivers before the press or even are not delivered at all, the sticky
key appears, making further gaming experience not appropriate. At the same time, streaming

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

requires a minimum amount of time for transferring data from client to server. Therefore, it is
difficult to switch to another network protocol under such constraints.
In this work, we present our approach to transferring data to the server. It was developed
within a Loudplay service. Still using the UDP protocol, we could suggest a solution tolerable
to network issues. Moreover, it is already integrated into the service, giving opportunity the
real users to be fully engaged in gaming.

2. Background
The need of sending all the necessary information and in the correct order leads us to the usage
of TCP protocol [13]. TCP works for keyboards without any inconsistencies. With mouses
and gamepads, it has less pleasant behavior. For example, mouse moves happen frequently
and produce a huge amount of data. TCP guarantee to deliver every package. Therefore, a
user could see mouse lags during the work, which is still not an appropriate solution for remote
gaming.
Nevertheless, many researchers have already developed streaming services employing the
advantages of UDP. “GamingAnywhere” [11] is open-source software that provides game
streaming. The authors of this software developed a solution that could be built on several
platforms. Although video and audio streaming performed well, the input handling has
limitations. This software has the problem described above: transferring control states.
Shea et al. [16] tried to analyze the impact of the network delays on the gaming experience.
The obtained results are ambiguous. Different games require different network tolerance. The
majority of first-person shooter (FPS) games are less playable when actions are delayed at a
little 100 ms. Massive multiplayer online (MMO) games have more tolerance (up to 500 ms)
for network delays because of the game mechanics. The conclusion is that for all games delays
should be at most 200 ms, which is difficult to reach due to the nature of the UDP protocol.
Kamarainen et al. [12] also described different limitations while streaming games: handling
user’s states, configuring audio and video for the requirements of users’ devices. They stated
that delay (the time difference between a user’s command input and the corresponding in-game
action) is a key factor that negatively affects the quality of user experience. However, their work
is mostly focused on support of video quality, rather than working with correct arrival of user’s
states.
As we noticed from the analysis of the literature related to the current state of the art, cloud
gaming became popular in both academia and industry. This fact was also investigated by
Chen et al. [7]. Such increased popularity opens new directions for future research, which will
lead to new opportunities in streaming software. As was mentioned before, we found our area
for investigation - the processing of control states of a user. This problem has already been
mentioned in different works, but not fully studied. Therefore, we take on the role of almost
pioneers in this unexplored area.

3. Proposed approach
X

X
se
ea

s
s
re
el
R

client ... 0 ... 1 ... server

Figure 1. Correct flow

Our goal is to deliver press and release messages for every button that was pressed in the
correct order as fast as possible. We showed schematically this process in Figure 1. Using UDP
we can achieve faster transmission but at the same time, we can face two issues. The first one

2
ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

is presented in Figure 2 where packages of press and release are delivered in reverse order. This
will lead us to the situation where the button will be pressed until another release message will
be processed on the server.

X
X

se
ea
ss
re

el
R
P
client ... 1 ... 0 ... server

Figure 2. Reordering flow

The second significant issue is shown in Figure 3 where the release packet was lost during the
transmission. The result of this situation is the same as the one described above.
X

X
se
ea

ss
re
el
R

P
client ... ... 1 ... server
×

Figure 3. Packet loss flow

Our proposed solution concludes in accumulating the state of the input devices and
periodically sending this state to the server. The period should be selected according to the
shortest period of action. The faster the user presses the keys the faster application should
send the data to the server. The reverse is also true. If the user does not press anything for a
significant amount of time, we are able not to send anything at all.
X

X
se
ea

ss
re
el
R

client ... 0000011111 ... server

Figure 4. Redundant flow

All the messages have the same structure of the package that is shown in Figure 5. The
maximum size of the package could be 65’635 bits. The timestamp is the time since the package
was sent to the server and represents the UNIX timestamp with UTC-0 timezone.
0 16 24 31

size type padding

timestamp

data
hhh
hhhh
h hh h
hhhh h
hhhhhhhhh
hhhh h
hhhhhhhhh
hhhh h hh
hhhh
hhhh hh h
hhhhh

Figure 5. General package structure

3
ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

3.1. Keyboard
In the case of a keyboard, we accumulate the state of all significant keys in a buffer, which
should be large enough to store all keys. We use a 256-bit buffer in the Loudplay service. All
the keys have two indicators: keycode and scancode. Keycodes represent the pressed symbol.
Using these indicators one has to deal with language synchronization. Therefore, we only focus
on scancodes since they represent the button itself. For example, Figure 6 shows the beginning
of the buffer. 8th bit represents the pressed backspace button which scancode is 0x20.
We use the cross-platform library SDL [6] for handling input events. This library has the
function “to get the current state of keyboard” - SDL GetKeyboardState. This function returns
the pointer to an array of states where 1 means that the key is pressed and 0 means that it is
not. Indexes of this array are obtained by using SDL Scancode values.
SDL Scancode values do not correlate with the platform scancodes. This means that we have
to map it to the server format directly. For example, in the case of windows OS we have to map
SDLK BACKSPACE to native 0x08. Result message is represented in Figure 7.
0 8 31

0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0...

Figure 6. Keyboard buffer

0 31

header

256 bit

Figure 7. Keyboard message

3.2. Mouse
The structure of the mouse message is shown in Figure 8. This structure is more complicated
than one of a keyboard. The buttons of a mouse are collected in a way similar to the one
of a keyboard. SDL has special functions for this activity - SDL GetRelativeMouseState or
SDL GetMouseState. Both of them are widely used and have the only difference. The first
function returns the mouse coordinates relative to the center of the screen. The other is relative
to the upper-left corner of the window. Mouse buttons are stored in a 24-bit variable to fit the
message layout. The values of the buttons provided by SDL cover only 3 buttons and are stored
in the first 3 bits of the region. We also support devices with only three buttons in our service.

4
ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

0 31

header
Buttons mode
x
y

Figure 8. Mouse message

0 8 16 31

header
padding index buttons
left stick x left stick y
right stick x right stick y
left trigger right trigger

Figure 9. Gamepad message

3.3. Gamepad
Figure 9 shows gamepad message. In our service, we give users the ability to play with
more than one device. Therefore, to define the exact number of gamepads for each client,
we need to store the identifier of the device itself. However, SDL does not provide the ability
to get all the states of the gamepad by using only one function. Therefore we have to use
SDL GameControllerGetButton and SDL GameControllerGetAxis for each button, stick, and
trigger. Common gamepads have around 16 buttons that fit in a 16-bit message, 2 sticks, and
2 triggers for 16 bits each.

4. Results and discussion

To evaluate the proposed approach, we run the tests using additional software “clumsy” [1] to
make our connection worse. We employed a wi-fi network of 2.4 GHz for tests and used 4 levels
of message drops:
(i) none: using standard network connection;
(ii) 0.1: increasing the probability of each message to be dropped by 0.1;
(iii) 0.25: increasing the probability of each message to be dropped by 0.25;
(iv) 0.5: increasing the probability of each message to be dropped by 0.5.
In table 1 one can see the results of our evaluation. Columns names represent the level of
message drops. Rows express the techniques: we have compared the performance of the naive
method with our approach. The values indicate the percentage of correctly registered key clicks
on a server out of 1000 clicks. From Table 1 one can notice that our proposed approach performs
better up to 5%.
Our approach shows an important advantage over currently existing naive methods. This
advantage concludes in an increased tolerance to package loss. The suggested solution can solve
one of the most disturbing situations in cloud gaming - the sticking of keys. However, our
method has some limitations.

5
ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

none 0.1 0.25 0.5

Naive method 96 95 93.5 90.3
Our approach 100 100 100 95.5

Table 1. Evaluation results

The first disadvantage of that approach is expressed in a significant increase in outgoing

traffic which could be very sensible on mobile networks.
The second disadvantage is that a mouse position sends with all the other states. That means
that in case of huge packet losses or case of big retransmission delay could occur a significant
mouse move gap. That could be important in FPS games. Another situation could occur during
the accumulation of user inputs. It is possible to click a key with that speed that will not be
registered by the software.

5. Conclusion and future work

In this paper, we presented our method which can handle the control states of users’ devices. Our
approach showed better results in comparison with naive method. The proposed method allows
researchers and software engineering practitioners to further investigate the gaming experience
opening new directions.
As was mentioned before, our work also has limitations. Therefore, our urgent goal will
be to decrease outgoing traffic and to handle the mouse positions messages. Other further
improvements could contain automatic calculating of the retransmission delay in case of the idle
state of a user. For example, if a user uses a gamepad the whole time we don’t need to send
keyboard and mouse messages. It is also possible to select some sort of gaming profile that will
define transmission delay.

Bibliography
[1] clumsy 0.2. https://jagt.github.io/clumsy/. Accessed: 2021-10-10.
[2] Gaming pc for the chosen one. https://www.loudplay.io/. Accessed: 2021-10-10.
[3] Number of active streamers on twitch worldwide from january 2018 to september 2021. https://www.
statista.com/statistics/746173/monthly-active-streamers-on-twitch/. Accessed: 2021-10-10.
[4] Number of youtube users worldwide from 2016 to 2021. https://www.statista.com/statistics/805656/
number-youtube-viewers-worldwide/. Accessed: 2021-10-10.
[5] Remote play. https://store.steampowered.com/remoteplay. Accessed: 2021-10-10.
[6] Simple directmedia layer. https://www.libsdl.org/. Accessed: 2021-10-10.
[7] Kuan-Ta Chen, Chun-Ying Huang, and Cheng-Hsin Hsu. Cloud gaming onward: Research opportunities and
outlook. In 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pages
1–4. IEEE, 2014.
[8] Terje Colbjørnsen. The streaming network: Conceptualizing distribution economy, technology, and power
in streaming media services. Convergence, page 1354856520966911, 2020.
[9] Julie Foertsch, Gregory Moses, John Strikwerda, and Mike Litzkow. Reversing the lecture/homework
paradigm using eteach® web-based streaming video software. Journal of Engineering Education,
91(3):267–274, 2002.
[10] Georgios Gousios, Dominik Safaric, and Joost Visser. Streaming software analytics. In 2016 IEEE/ACM
2nd International Workshop on Big Data Software Engineering (BIGDSE), pages 8–11. IEEE, 2016.
[11] Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. Gaminganywhere: An open cloud
gaming system. In Proceedings of the 4th ACM Multimedia Systems Conference, MMSys ’13, page 36–47,
New York, NY, USA, 2013. Association for Computing Machinery.
[12] Teemu Kämäräinen, Matti Siekkinen, Yu Xiao, and Antti Ylä-Jääski. Towards pervasive and mobile gaming
with distributed cloud infrastructure. In 2014 13th Annual Workshop on Network and Systems Support
for Games, pages 1–6. IEEE, 2014.
[13] Santosh Kumar and Sonam Rai. Survey on transport layer protocols: Tcp & udp. International Journal of
Computer Applications, 46(7):20–25, 2012.

6
ITTCS 2021 IOP Publishing
Journal of Physics: Conference Series 2134 (2021) 012018 doi:10.1088/1742-6596/2134/1/012018

[14] Andrew S Liteplo, Vicki E Noble, and Ben HC Attwood. Real-time video streaming of sonographic clips
using domestic internet networks and free videoconferencing software. Journal of Ultrasound in Medicine,
30(11):1459–1466, 2011.
[15] Craig Partridge and Stephen Pink. A faster udp (user datagram protocol). IEEE/ACM Transactions on
Networking, 1(4):429–440, 1993.
[16] Ryan Shea, Jiangchuan Liu, Edith C.-H. Ngai, and Yong Cui. Cloud gaming: architecture and performance.
IEEE Network, 27(4):16–21, 2013.