1
GENERALIZED SOFTWARE APPLICATION FOR OPERATION OF
A 3D VEHICLE IN AIR, WATER AND LAND
Diogo Peres(1) (2), António Raimundo (1) (2), Nuno Santos(1) (2), Pedro Sebastião(1) (2), Nuno Souto(1) (2) , Alexandre Almeida(1) (2)
(1)
ISCTE-Instituto Universitário de Lisboa, DCTI, Avenida das Forças Armadas, 1649-026 Lisboa
(2 )
Instituto de Telecomunicações, Av. Rovisco Pais 1, 1049-001Lisboa
email: drbps@iscte.pt; aslro@iscte.pt; nmass@iscte.pt;
pedro.sebastiao@iscte.pt; nuno.souto@iscte.pt; alexandre.almeida@iscte.pt
Abstract — The unmanned vehicles (UV) and its applications
are growing exponentially. Using the radio control is the most
common way to control these types of vehicles for being a
simple and cheap method to control an UV. However, it doesn’t
have a visual interface that allows the user to see the vehicle
information such as, the battery status, speed, distance,
geolocation, etc. To help with this problem, some mobile and
desktop applications have been developed. To connect the
device to the vehicle, dongles are commonly used, they can
connect by radio, Bluetooth or Wi-Fi. In most cases, these
technologies don’t allow the user to control at long distances,
beyond line-of-sight, and the applications are focused to use
mostly on multi-copters, and most of the times, they allow the
connection of only a vehicle at each time.
The purpose of this work is to study the reliability of a software
be able to control various types of vehicles, such as aerial, land
and water vehicles. This software allow the user to connect
multiple vehicles to the same device at the same time, easily
change the vehicle assigned to control, by using mobile
networks to perform the communication between the software
and vehicle. In this way, it will be possible to connect a vehicle
3D, which is a hybrid vehicle capable of moving in water, land
and air environments, allowing the user to control the vehicle at
long distances with video feedback.
Key-words: vehicle 3D, Android, unmanned vehicles, multi-
vehicle, mobile networks.
1. INTRODUCTION
In recent years, the use of unmanned vehicles has grown
quickly. The technology was developed primarily to use in
the military area. The ability to use them in dangerous
missions without risking human lives was the main factor
that led to its creation. The success of this technology
aroused the curiosity of people who quickly realized the
potential of these technologies. Nowadays, unmanned
vehicles can do all kinds of tasks. In the military field, they
can help with search and rescue operations, surveillance,
explosives disarming, pursuit operations, delivering supplies
to remote and/or inaccessible regions, border patrol
missions, crowd monitoring, and so on [1]. Its great utilities
have led to a fast growth of the UV market which has led to
a growth in the appearance of applications capable of
controlling and monitoring them, usually called ground
control stations (GCS). Since one of the markets that has
been growing the most is the UAV market, more precisely
those of multi-copters, most applications are focused to use
with multi-copters. However, this fast grown led to the need
of new applications capable of controlling new types of
UV’s. This paper was focused in developing an application
not only capable of controlling and monitoring various types
of UV’s as well as a hybrid 3D vehicle, a vehicle capable of
moving in three dimensions, water, land and air.
2. 3D VEHICLE
To simulate a vehicle capable of moving in three
environments, it was used a program called SITL, capable of
simulating various types of UVs. This program was used to
simulate an unmanned aerial vehicle (UAV), an unmanned
ground vehicle (UGV) and an unmanned surface vehicle
(USV).
It was chosen to test in three vehicles separately to prepare
and test the application with a single vehicle connected at a
time and lastly, connect the three vehicles simultaneously to
simulate a 3D vehicle.
A real 3D hybrid vehicle was not construct since it would
took a long time to build and the main focus of this paper is
the application to control a hybrid vehicle.
In addition, the controller boards are not prepared for 3D
vehicles, so it was decided to construct three vehicles and in
case the user wants to make a 3D vehicle it will be only
necessary to put together the three electronics in a single
frame.
The application was constructed in a way that allows the
user to change between vehicles easily, in a 3D vehicle this
action corresponds to the exchange between the different
driving modes. Two real vehicles were also constructed, a
hexacopter and a scaled model of a car, to test the
application in a real scenario.
3. HARDWARE USED
Besides the common electronics used in a UV, such as: the
motors and servos, the electronics speed controllers to
regulate the amount of power passing to the motors and the
battery, it was used other hardware to be able to connect the
vehicle to the network and communicate via wireless
networks with the mobile application. Figure 1 shows the
technical architecture of the solution proposed. The
Raspberry Pi and the Controller Board were placed in the

vehicle and communicate with the server. The server
communicates also with the mobile application to exchange
the messages coming from and to the vehicle
Raspberry Pi Mobile
Controller Application
MAVProxy
Board
Java Program Server
Figure 1 - Technical Architecture
1) Controller Board
Controller boards for UVs are generally referred as flight
controllers because they are mostly used for aerial vehicles,
responsible for managing the flight of multi-rotor aircrafts
and airplanes. Its purpose is to stabilize the aircraft during
flight, to do so, it takes the signals from on-board
gyroscopes and passes them to the processor, which
processes signals according to the user’s selected firmware
(corresponding to its type of vehicle) and passes the control
signals to the installed ESCs to make fine adjustments to the
motors rotational speed which in-turn stabilizes the aircraft.
Controllers boards also uses signals from radio or other
sources like Bluetooth or Wi-Fi modules and passes these
signals (e.g. aileron, elevator, throttle and rudder) together
with stabilization signals to the processor, which once
processed send them to the ESCs to turn and control the
vehicle orientation. There are several controller boards
available in the market, although, it was necessary a
controller board with advance functionalities like GPS and
auto guide. One of the most known flight controller is
Ardupilot Mega (APM), and its evolution Pixhawk which
was chosen as the main controller to be used because it has a
better performance, uses an open-source software and is also
suitable for most types of radio control vehicles: fixed
wings, copters, cars and boats, depending on the firmware
loaded
To communicate with external systems such as ground
control stations this controller board uses a messaging
protocol called MAVLink.
2) Raspberry Pi
To connect the vehicle to the network its necessary a way to
forward the messages coming from the controller board
through Wi-Fi or 3G/4G, to do so it will be used a raspberry
pi.
Raspberry pi was chosen for being a tiny and affordable
computer, capable of running Linux, with a big community
of developers, and capable of connecting to real world
objects with the help of GPIO pins, USB ports, a HDMI port
and audio jack.
It was also used a raspberry camera module and installed a
collection of drivers and API called Video4Linux for
supporting realtime video capture on Linux systems.
2
The camera was used so the application could receive video
feedback and allow the user to see were it’s going in case the
vehicle is out off line of sight. The raspberry pi was placed
in the vehicle and connected to the controller board and had
a 3G/4G dongle to be able to connect to the internet.
4. SETTING UP
To communicate and prepare the hardware to deal with the
different types of vehicles and communicate between
different units it was used several software programs.
1) Mission Planner
To make the controller board functional it was needed to
load a firmware. There are several GCS available in the
market capable of doing it. For this paper, it was chosen
Mission Planner for being a simple, free and widely used
GCS that is available for Windows. After installing it, the
controller board should be connected to the computer with
the help of a micro USB cable and installed the firmware
according to the type of vehicle pretended (ArduRover for
cars and boats, ArduPlane for fixed-wings, ArduCopter for
Helicopters, ArduCopter Quad for quadcopters, ArduCopter
Hexa for hexacopters, etc) [2].
2) MAVLink
Micro Air Vehicle Link (MAVLink) is a communication
protocol used mostly for communications between a GCS
and UVs. It can be used to transmit the altitude of the
vehicle, speed, battery status, its GPS location, and other
information. MAVLink can pack C-structs over serial
channels with high efficiency and send these packets to the
GCS. It was extensively tested on various controller boards
like the PX4, Pixhawk and APM. In other words the
MAVLink message is just a stream of bytes encoded by the
GCS to send to the controller board or vice-versa. The
stream of bytes can be sent by USB serial or telemetry (both
cannot be used at the same time, if the both are plugged in,
USB is given preference and Telemetry is ignored).
Table 1 – MAVLink Message packet structure
Content STX LEN SEQ SYS COMP MSG PAYLOAD CKA CKB
Byte Ind 0 1 2 3 4 5 6 to (n+6) n+7 n+8
A MAVLink packet has the structure described in Table 1.
The first 6 bytes are the message’s header, being the first
one to indicate the start of a new MAVLink packet, the
second byte has the message’s length, third byte is the
sequence number, fourth corresponds to the system ID and
indicates which system is sending the message, allowing to
distinguish each vehicle or GCS in the same network, the
fifth byte has the component ID and allows to identify
different components on the same system, such as two GCS
running in the same machine and the last byte in the header

corresponds to the message’s ID, which allows to decode the
payload correctly. The Payload, as shown in Table 1 is the
actual data and the last two bytes are the checksum which
the software uses to check if the message is valid and not
corrupted. A MAVLink message has a minimum length of 8
bytes, typically for acknowledgement packets (ACK)
without payload and a maximum length of 263 for full
payload [3].
3) MAVProxy
MAVProxy is a command line GCS with the ability to
forward the messages from the UV controller board over the
network via UDP to other GCSs or devices. It is written
100% in Python and it’s an open source software. This
software will be used in the Raspberry Pi to be able to read
the messages coming from the controller board and forward
them through the network to other devices.
4) SITL
To help the tasks of this work it was used a program to
simulate the controller board in the vehicle (able to simulate
APM or Pixhawk). The simulator is called SITL (Software
In The Loop) and is based on Arducopter, allowing to run
Plane, Copter or Rover without any hardware. SITL is
especially useful since some malfunctions in the program
while in the development phase can cause the vehicle to
crash. With SITL, the program can be first tested in the
simulator and then, when stable, passed to the real vehicle.
This simulator can be run either on Windows or Linux.
Since SITL will emulate the controller board it will be still
needed a way to communicate with it, to do this it will be
installed MAVProxy alongside with SITL.
5) Server
One of the main challenges of this work was to deal with the
network address translation (NAT). The public IP of a
device changes according to the network that it’s in and due
to this fact the application doesn’t know the IP of its vehicles
and vice-versa. To overcome this problem the
communications were done using a server.
Each time a vehicle or an application is turned on, it will
connect to a server that will authenticate the device and
register its IP, the server will also redirect the messages from
the application to its vehicles and vice versa.
The server stores the user information such as the username,
password and the UV’s corresponding to each user.
6) Vehicle Java Program
It was also developed a java application that was exported to
a jar file. This jar file runs in the raspberry pi placed in the
vehicle and it’s started each time the vehicle its turn on.
These program is responsible for connecting the vehicle to
the server. The applications stores in a .txt file an individual
ID and token that are different for each vehicle and they are
used by the server to authenticate the vehicle to increase its
3
security. The application redirects to the server all the
packages that are coming from MAVProxy and vice-versa
(Figure 1).
5. MOBILE APPLICATION
The mobile application was developed in android studio. It
was opted for Android because it has many users on a global
scale, it’s free and is a native technology, having a better
performance in the devices using android. The minimum
version targeted was android Jelly Bean 4.1.xx since it will
run on 97,5% of android phones (considering the time that
the paper was written).
To be able to decode the MAVLink messages it was used a
library called “MAVLink Java” which decodes the bytes
with the help of xml files and generates Java code.
The mobile application called ‘Mobile Control Station’ starts
in a login screen which receives a username and a password
to authenticate in the server. If the server responds with a
successful message the user is redirected to a main screen
with the vehicle list, displaying the vehicles ready to be
controlled, so the user can select one.
The main screen has two tabs at the top. In the first one it’s
possible to manually control the vehicle by using two
joysticks and in the other tab, called mission tab, it’s
possible to monitor and control the vehicle by sending
waypoints.
The main screen has also a menu with five options that will
be explained later.
A. Controller Screen
The controller tab redirects to a controller screen which is
used to check the vehicle information’s and manually
control it.
Figure 2 – Controller Screen
There are several features in the controller screen, such as
the Arm/disarm button. When the vehicle is disarmed it will
not respond to the commands sent by the user. The vehicle
has to be first armed in order to control it.

Mode Button, in Figure 2 corresponds to the button
displayed as “stabilize”. Once this button is clicked it will
display a popup to allow the user to change the guiding
mode of the vehicle. Each type of vehicle has its own modes
that can be for example, the stabilize mode, only existent in
flying vehicles, corresponding to the manual mode in the
rover firmware. This mode allows the user to control the
vehicle by using the joysticks in the application.
Auto mode. Once the user sends a set of waypoints to the
vehicle and changes to auto mode, the vehicle will start the
mission and go through the waypoints sent autonomously
until it reaches the final one where it will remain still.
Guided mode. If the vehicle is in guided mode the user can
send a waypoint to the vehicle which will move immediately
to the waypoint sent.
RTL mode (Return to Home). As the name suggests, if this
mode is enabled the vehicle will return to home which by
default is the point where it was turned on.
Some modes require the GPS module to be installed in the
controller board to work.
Controller screen has also a battery level indicator and a
connection status indicator. This indicator displays the
Estimated RTT (Estimated round trip time of the messages)
̂ ̂
𝑅𝑇𝑇𝑛𝑒𝑤 = (∝ 𝑅𝑇𝑇𝑜𝑙𝑑 ) + ((1−∝)𝑅𝑇𝑇𝑛𝑒𝑤𝑠𝑎𝑚𝑝𝑙𝑒 ) (1)
The value for variable ∝ (that could go from 0 to 1) was 0 to
make the new ̂ 𝑅𝑇𝑇 the value of the last round trip time.
In the grey top bar there is placed three labels, the first one
indicates the height (in meters) of the vehicle. The second,
labeled as ‘VS’, indicates the vertical speed in meters per
second, and the last one, labeled as ‘HS’, indicates the
horizontal speed, in meters per second as well.
The Blue button at the bottom center enables and disables
the joysticks that will enable to manually control the vehicle.
Controller screen has also the video feedback overlaid with
an HUD (Heads Up Display, displayed at green) with the
purpose of helping the user to have a better perception of the
vehicle’s direction and position.
B. Missions Screen
As the controller screen, the missions screen also displays
the vehicle’s information, like the altitude, speed, battery
status, connection status, has an arm/disarm and a mode
button and a HUD. The mission tab allows the user to
monitor one or more drones and send waypoints so they
could move autonomously.
4
Figure 3 – Missions Screen
There are several buttons in the missions screen.
1. There is a takeoff button, displayed in Figure 3. In this
button, the user can click to edit the desired altitude
or/and takeoff, the vehicle will then takeoff
autonomously and stay or circle the desired altitude.
2. Request waypoints button gets the last waypoints sent to
the controller board, which are the ones the vehicle will
travel once the user changes the vehicle mode to auto.
3. Edit waypoints button enables the waypoint editing, if
this button is clicked, its label changes to stop editing
and two more buttons appear, one to clear all the
waypoints and one to send the waypoints created to the
controlled board. To create waypoints the user only has
to have the edit waypoints option enabled and click in the
map in the desired location. The user can then click in a
waypoint to edit its altitude. The vehicle will move to the
altitude introduced in the waypoint, which by default is
10 meters, ground and surface UV’s will ignore this
parameter. Once the user finishes editing the waypoints,
the send button must be clicked to send the waypoints to
the controller board and change the vehicle mode to auto
to start the autonomous mission.
The vehicle will move through the waypoints in the order
they were introduced, the application helps the user by
showing the UV’s path while editing the waypoints.
The user can make a long press in a location in the map and
the application will automatically change the vehicle mode
to guided and send it to the location clicked while
maintaining its altitude.
C. Side Menu
The three dots at the top left of the screen opens a side menu
with five options. These options are:
1) UV’s selection
The first option displays a popup list with the vehicles
connected to the application, an image corresponding to the
type of vehicle and a unique identifier.
In this list the user can change the vehicle selected to
control, this is necessary since the application it’s able to
 5

display and monitor several vehicles at a time but it can only

control a single one, which is the one selected by the user.
2) Parameters
This option requests the controller board all the parameters
and their values. Once all the parameters are received, it
opens a screen with all the parameters so the user can see or
edit their values. Parameters allows the user to change the
vehicle configurations. An example of a parameter is the
vehicle sensibility to the controlling commands.
3) Settings
The Settings Screen have the settings for each axis in the
joystick. In this screen is possible to revert the controls,
define the minimum and maximum of the axis and trim the
middle value up or down like in a real radio controller.
4) Statistics
The Statistics option opens a statistics screen where there are
displayed four graphs. The first one indicates the battery
level over time. The second graph has two lines, a blue one
for the Vertical Speed and a brown line for the Horizontal
Speed, indicating the variation of speed over time. The third
graph has the number of messages received and a label with
the average number of messages per second as well as the
total number of messages received since the application was
initialized.
𝐷𝑒𝑣 epresents the deviation, 𝐷𝑒𝑣𝑛𝑒𝑤 is the new value for
deviation and 𝐷𝑒𝑣𝑜𝑙𝑑 is the estimated old deviation value.
If a response message is not received within the timeout, it is
given as lost. The percentage of received messages, RM, is
given by:
𝑇𝑝𝑚𝑠−𝑇𝑚𝑙
𝑅𝑀 = × 100 (4)
𝑇𝑝𝑚𝑠
Tpms is the total ping messages sent and Tml is the total of
messages lost.
5) Logout
Lastly the logout option button sends a TCP message to
disconnect the user.
6. SIMULATION AND EXPERIMENTAL RESULTS
To test the application several tests were done using SITL
and using real unmanned vehicles. First the application was
tested with a vehicle at a time, next it was tested with two
and finally with three vehicles connected to the same user at
the same time, simulating a 3D vehicle.
The 3D Vehicle will appear in the vehicle list as three
separated vehicles and the user will only need to select the
type of vehicle it wants to control (Figure 5).

Figure 5 – Vehicle list

Figure 4 – Statistics screen.

The last graph displays the RTT over time and the packet
loss. The RTT is calculated with the equation (1). To get the
packet loss it’s necessary to calculate the timeout for a
packet, which is calculated as

̂ + 4 × 𝐷𝑒𝑣
𝑇𝑖𝑚𝑒𝑜𝑢𝑡 = 𝑅𝑇𝑇 (2)

and

̂ | (3)
𝐷𝑒𝑣𝑛𝑒𝑤 = (1−∝) 𝐷𝑒𝑣𝑜𝑙𝑑 + ∝ |𝑁𝑒𝑤𝑅𝑇𝑇𝑛𝑒𝑤𝑠𝑎𝑚𝑝𝑙𝑒 − 𝑅𝑇𝑇 Figure 6 – Multi vehicle monitoring
 6

It was tested three types of firmware. Rover, Copter and For future work, it could be developed a firmware to the
Plane. Tested both on controlling and monitoring. The Plane controller board able to deal with a 3D vehicle, so that it
firmware was only tested in the simulator since it was not wasn’t necessary to have three separated units together.
constructed a fixed-wing. Another suggestion is to improve the delay, i.e., minimizing
The car, boat and copter were tested in simulation and with the delay time in the video streaming.
real vehicles.
The modes tested where: ACKNOWLEDGEMENTS
• Guided (Copter, Plane and Rover);
• Auto (Copter, Plane and Rover); The authors would like to thank Instituto de Telecomu-
• Stabilize (Copter and Plane); nicações, (Portuguese R&D Centre) and ISCTE-University
• Position Hold (Copter); Institute of Lisbon, by providing funds for the acquisition of
• Manual (Rover); all the material needed to carry out this project. Without
• RTL (Copter and Plane); these funds the work would not be possible.
The application was continuously being tested to ensure all
the bugs were tested and maximize the application’s REFERENCES
performance.
The statistics screen allowed to gather information. One of [1] C. E. Nehme, "Modeling human supervisory control in
the information collected was the average RTT for the heterogeneous unmanned vehicle systems,"
controller messages, which was tested several times using Massachusetts, 2009.
different scenarios, to construct Table 2. It was tested with
[2] "Mission Planner Overview" [Online]. Available:
and without video feedback, the RTT increase with video
http://ardupilot.org/planner/docs/mission-planner-
because the video streaming requires more processing
overview.html. [Accessed 12 August 2016].
power, has a higher bit rate than the controlling messages
(MAVLink) and requires more Bandwidth. [3] "MAVLink Micro Air Vehicle Communication
Protocol," [Online]. Available:
Table 2 – Controlling messages RTT http://qgroundcontrol.org/mavlink/start. [Accessed 29
August 2016].
Video 1 vehicle 2 vehicles 3 vehicles [4] Szablewski, D. (Ed.),“HTML5 live video streaming via
Feedback [ms] [ms] [ms] websockets,” 11 September 2013. [Online]. Available:
No 127 134 142 http://phoboslab.org/log/2013/09/html5-live-video-
Yes 172 188 210 streaming-via-websockets. [Accessed em 6 December
2016].
The approached used to have video feedback was video
streaming via websockets. First a multimedia framework
called ffmpeg need to be installed in the raspberry pi in order
to handle the video coming from the camera. This
framework will encode the video to MPEG and send it to the
server via HTTP. The server will need to have installed a
JavaScript framework called nodejs to run a nodejs script
that will distribute the MPEG stream via Websockets to the
connected clients that will decode it to reproduce the video
[4].
To test the connection a HTML and Javascript files were
downloaded from jsmpeg project in GitHub. These files
were loaded and placed in a webview in the application.
They connect the application to the server via a websocket
and display the image in the webview.

7. CONCLUSIONS AND FUTURE WORK

As a main conclusion, it was successfully developed an

application able for control and monitoring of one or more
vehicles as well as a 3D vehicle, using mobile networks to
communicate and receive video feedback. Allowing the
control at long distances. The application provides the user
an affordable and versatile way to control in real time
various types of vehicles.