Monitoring and Controlling Virtual and Hybrid Test Benches in the
Cloud
A. Lars Stockmann1, B. Jan Wasmund1, C. Sören Reglitz1
1: dSPACE GmbH, Rathenaustraße 26 33102 Paderborn
Abstract: Efficiently operating a test bench is crucial for
the success of small- and large-scale projects in the aero-
space industry. Traditionally, benches are operated in a lo-
cal and often isolated environment, which requires the staff
to be on site. Lockdown situations have made this difficult
recently, and it became clear that it must also be possible
to operate the test bench remotely. Making this remote ac-
cess safe, secure, and accessible poses a major challenge
for suppliers and OEMs alike. A novel approach makes use
of the modularity and openness of test systems following
the ideas of the Virtual and Hybrid Testing Next Genera-
tion (VHTNG) research project – a project that aims to cre-
ate a standard for a communication infrastructure by
providing standardized transport mechanisms required for
interoperability. The approach transfers the VHTNG inter-
faces for unified monitoring, which are currently applied
only in local environments, to the cloud. The reasoning be-
hind this is that engineers will eventually perform more
tests in the cloud and therefore require location-independ-
ent test bench services. The approach is based on a dedi-
cated architecture and a technology stack that enable users
to monitor multiple test sites that can consist of one or more
test benches via common web browsers. However, the con-
trol aspect has not been discussed. Therefore, this paper ex-
plores the possibility of using this approach to control test
benches, too, without sacrificing any of the security-related
core principles, that is to say, no full access to the test bench
and no requirement to expose the test systems directly to
the Internet. The approach is a direct advancement and is
based on the same technology stack (i.e., VHTNG and the
Elastic Stack). Just like the previous paper, this work fea-
tures a dedicated requirements analysis and presents the
prototype that was used for evaluation.
Keywords: testing, cloud, remote control
1. Introduction
Operating a test bench can be a challenging task. The tester
must always keep an eye on all the involved laboratory test
means (LTM) and take appropriate actions in case of errors.
In a traditional monolithic test bench, the test system sup-
plier is responsible for providing the proper tooling for this
task. However, in a heterogeneous test bench, where sys-
tems originate from different vendors, a tester must be fa-
miliar with all the different ways to operate those systems
using their respective tools. This is not only inconvenient
but can also lead to confusion and requires dedicated train-
ing for each system. Moreover, unified access to the
ETTC 2021– European Test & Telemetry Conference
different test systems of the bench to monitor and control
the complete test environment from a single frontend is not
possible because multiple proprietary interfaces need to be
considered.
To improve on this, the Virtual and Hybrid Testing Next
Generation (VHTNG) research project is developing a ge-
neric and modular architecture for an open test environ-
ment. The goal is to prepare a set of open standards for the
interfaces required to build this architecture [1]. This in-
cludes data models and protocol stacks to monitor and con-
trol such an environment.
Until recently, the VHTNG project has focused primarily
on the local communication in the test bench. However, it
has always been ensured that the data models are generic
and not limited to a local environment. During the Covid-
19 pandemic, it became abundantly clear that having safe
and efficient remote access to the bench is a must-have for
future test benches. It is a key enabler for flexible testing
and efficiently conducting large test campaigns with test
benches allocated on different sites.
However, providing remote access to monitor and control
a test bench poses additional challenges, with safety and
security being the most critical. Adversaries gaining full
access to a test bench could sabotage it to a degree that puts
people’s health at risk. Also, monitoring information, such
as log data, might contain sensitive information. This could
include confidential information about the stage of devel-
opment, which must not be leaked to competitors in the in-
dustry. Other aspects are maintainability, usability, and, of
course, costs.
Therefore, VHTNG concepts and models have been ex-
panded to create a secure monitoring solution that can be
deployed in the cloud [2]. This solution makes it possible
to monitor an arbitrary number of test sites, each consisting
of several test benches. In contrast to existing approaches,
it does not impose any additional requirements on an LTM.
This means that it is not required to install additional soft-
ware, nor is there a need to directly connect equipment to
the Internet. This makes the new approach much easier to
set up in companies with strict IT restrictions and more se-
cure by design. Furthermore, it can leverage the vendor-
independent nature of VHTNG to include any device if the
respective device implements the VHTNG Status Monitor-
ing interface. This includes completely virtualized equip-
ment, which itself may be hosted in the cloud.
The approach is based on the latest, yet well-established
web technologies and standards. From an end user perspec-
tive, the approach also supports handheld devices and only
requires a common web browser.
This paper advances the previously presented approach by
adding the possibility to control the test systems. It again
leverages the data models and concepts from VHTNG,
while at the same time ensuring that the security principles
of the original approach are not impacted.
The paper is structured as follows: The next section pre-
sents a requirements analysis for controlling test benches
remotely, which is based on the requirements analysis al-
ready presented for monitoring hybrid test benches in the
cloud [2]. This analysis forms the basis for the main con-
cept, which is presented subsequently, followed by a de-
scription of the prototype that was used for evaluation. Af-
ter that, existing solutions and related approaches are dis-
cussed. The paper closes with a conclusion and outlook.
2. Requirements Analysis
To efficiently control a test bench, it is a prerequisite that
its components provide information regarding their current
state and potential errors. In the VHTNG project, these
components are referred to as modules. It is assumed that
every module that is connected to the communication in-
frastructure (see [3]) exposes its status monitoring interface
and its control interface. The control interface depends di-
rectly on the existence of the status monitoring inter-
face [4]. Consequently, remote control inherits all require-
ments from the remote monitoring of test benches, which
were discussed extensively by Stockmann et al. [2]. The
following subsection will summarize these requirements.
2.1 Requirements for Remote Monitoring
The first requirement (REQ1) specified in [2] defines the
minimal set of information relevant for monitoring. It con-
sists of:
1. Status information
• Information about the type, health status, and
state of a module (see Figure 1)
• Information that a module is ‘alive’
• Hierarchy of the test bench
2. Log information
Together, the status and log information represent what is
hereafter referred to as monitoring information.
Children
* «enumeration»
Health
MonitoredEntity
Unspecified
+ Name: string {id} Ok
+ Type: TypeId Warning
+ Health: Health Error
+ State: State Critical
Figure 1. UML diagram of the VHTNG model con-
taining the essential status information (cf. [2])
ETTC 2021– European Test & Telemetry Conference
Requirements 2 to 10 are related to accessing this infor-
mation:
REQ2 demands location transparency
REQ3 demands support for heterogeneous test benches
REQ4 demands support for real and virtual equipment or
a mix of both in a hybrid test environment
REQ5 demands support for cryptographically secure
data exchange
REQ6 demands support for an authentication and author-
ization infrastructure (AAI)
REQ7 demands that no LTM require a direct Internet
connection
REQ8 forbids full access to the test bench
REQ9 demands that the mechanism not rely on VPNs or
special firewall settings (to be easily deployable in
a company environment)
REQ10 demands that the user interface (UI) be adapted to
the end user’s device
The next requirements ensure that the mechanism scales
well and is generally robust and cost-effective:
REQ11 demands that the mechanism itself not impose any
limits on the number of test sites, test benches, and
their contained entities.
REQ12 demands that the mechanism can adapt to the size
and budget of any company
REQ13 demands that the mechanism must support com-
putationally constrained, low-bandwidth environ-
ments on the end user’s side
REQ14 demands that the mechanism is reliably available
around the clock
It must be noted that remote monitoring being the base for
remote control is not the only reason for which all these
requirements are valid. In fact, even if one were to consider
having a “blind” remote control, that is, a control without
monitoring, the requirements would still be relevant. For
example, control should be possible independent of the ac-
tual location of a test site (REQ2) and it should not be re-
quired to have the LTM exposed to the Internet to start a
simulation (REQ7). Also, a complex firewall or VPN setup
should not be required (REQ9) just to load a configuration.
Still, there are requirements that are control specific. These
are discussed in the following sub section.
2.2 Additional Requirements for Remote Control
Apart from those mentioned in the previous section, there
are two additional requirements that are exclusive for the
control aspect. Firstly, following VHTNG and the analysis
conducted by Stockmann and Himmler [4], this paper as-
sumes that the key piece of information relevant for control
is the command. Note that in the context of a test bench,
multiple aspects can be handled using commands. This pa-
per, however, only considers control commands, which are
used to perform certain tasks on the test equipment, such as
loading a configuration or starting a simulation.
Traditionally, commands are implemented in the form of
remote procedure calls (RPC, cf. [5]) in a request/response
fashion. In addition, commands can have parameters to
convey extra information. Remote controlling the modules
of a test bench should “feel” just like on-site. Thus, REQ15
demands that the approach supports a command-based con-
trol paradigm.
Having a way to remotely control the modules of a test
bench always entails certain risks. No matter how well the
AAI is set up (see REQ6), accounts can become compro-
Back end
Cloud
mised or security vulnerabilities can be uncovered. As of Web server and reverse proxy
May 2021, over 1,000 entries are listed in the CVE data-
base1 in conjunction with authorization and over 1,800 en-
tries with authentication. Therefore, the mechanism must
provide a reliable means of checking and, if applicable, Get static UI Push site lo- Query
overriding/blocking the remote control (REQ16). Push cation, sta- and re-
com- Query status tus monitor- ceive
mands monitoring & ing & log com-
3. Cloud-Based Monitoring and Control Concept log updates updates mands
There are well-known traditional approaches to controlling
a test bench remotely, such as terminal-based access, for Gateway
START()
example, via secure shell (SSH) or remote desktop. They STOP()
START()
all fail to meet the requirements of status monitoring, which STOP()

has already been discussed [2]. In summary, there are three
main problems:
1. They offer full access to the bench, which is not
desired as per REQ8
2. Because it is too dangerous to expose such ser-
vices directly, they must be shielded using, for ex-
ample, a virtual private network (VPN) failing to
comply with REQ9
3. The user interface (UI) does not adapt to the end
user device, failing to comply with REQ10
It has been found that a better approach is to let the cloud
act as a shield between the user and the test bench (or test
site). This is also true for control. Here, it must be made
sure that no direct connection to the bench is required
(cf. REQ7).
The approach presented in [2] achieves this using an Elas-
ticsearch service. It retrieves the site and status monitoring
information as a log message from a dedicated gateway and
answers requests for status monitoring information from
end user devices. This way, the end user device never has
a direct connection to the test bench. Likewise, the gateway
at the test site does not know the end user’s device. This
fact can be exploited and the paradigm can be inverted to
realize the control interface.
The idea is to have the end user device and the gateway
switch roles. In other words, the end user device pushes
commands (as per REQ15) in the form of a structured log
message into the Elasticsearch database and the gateway
queries them. Naturally, this does not fail to meet any of
the requirements presented in section 2.1 as the mechanism
itself does not change. The only change is that the gateway
now acts as a controller (instead of just an observer) inside
the local test bench. An overview of the additions to the
original concept is depicted in Figure 2.
VHTNG communication infrastructure
Front end
Test site
Figure 2: Overview of cloud-based monitoring &
control
This approach also fulfills REQ16, as it can serve as the
central point to inspect incoming commands before for-
warding them. Also, the remote-control interface can be
fully deactivated should it be necessary by simply disabling
command processing on the gateway. Nothing needs to
change regarding the other modules.
To evaluate the concept, a prototype has been imple-
mented, which is presented in the next section.
4. Prototype and Evaluation
The prototype is based on the one presented in [2], which
has been extended to support the execution of commands.
It uses the same virtual machine deployed in Microsoft Az-
ure2 to host Elasticsearch. Nginx3 is used to serve the static
content of the frontend and to act as a reverse proxy, thus
allowing both the gateway and the clients to use standard
http(s) ports.
Figure 3 illustrates the current frontend, which is based on
a single-page application. It provides an overview of all test
sites (worldwide).

1 3
CVE® Program website: https://cve.mitre.org Nginx web site: https://www.nginx.com
2
Microsoft Azure Cloud website: azure.microsoft.com
ETTC 2021– European Test & Telemetry Conference

Figure 3: Screenshots of the remote monitoring front
end on a smartphone
Here, users can hover over a test site and select a contained
test bench. This opens the test bench overlay, presenting
the individual modules of a test bench. Users can then open
a context menu from each of the modules, allowing them
to send the control commands. The frontend uses the status
information of a module to identify which commands are
currently allowed according the module’s state. The other
commands are disabled and consequently grayed out, as
can be seen in Figure 4.
Figure 4: Screenshots of the context menu containing
currently available control commands
The control command itself is encoded into a structured log
message. It includes the site name, the command name, the
target, and a timestamp when the command was sent. The
log message is pushed into the Elasticsearch database in the
cloud. In the current version of the prototype, the client de-
vice generates the required Elasticsearch queries directly,
effectively bypassing the back end depicted at the top left
of Figure 2. In a future version, the back end will provide a
simple, more convenient API that is better suited for low-
ETTC 2021– European Test & Telemetry Conference
powered end user devices. The goal is to make the approach
more modular and efficient.
The gateway has been extended in two ways, as shown in
Figure 5. First, it now implements an Elasticsearch client
to query the command messages. Second, it now acts as a
VHTNG control module, so it can forward the decoded
control commands to the respective modules in the local
test bench.
Gateway Module
gRPC
Collector
and HTTP(S)
serializer
VHTNG
Adapter Elas-
START()
ticsearch STOP() HTTP(S)
client
START()
STOP()
VHTNG communication infrastructure
Figure 5: Gateway with Elasticsearch client
While evaluating the prototype, it has been found that the
concept works well. A major benefit is that all of the con-
trol tasks performed remotely are logged automatically.
There are, however, also some minor drawbacks and chal-
lenges that must be overcome in the future.
One problem is an elevated risk of a race condition due to
the overall latency and loose coupling between the local
bench, Elasticsearch, and the client device. This occurs
when a command requires a module to be in a dedicated
state. For example, consider a hypothetical STOP com-
mand that requires the targeted module to be in a
STARTED state. The frontend verifies the current state of
a module and only allows a user to send commands that are
allowed in the current state. However, it is possible that the
front end might not have received the notification that a
module state has changed (e.g., in this case, the module
may have changed to a STOPPED state due to an error).
The front end consequently allows the user to send com-
mands that are in fact prohibited for the actual (local) state.
This results in the command seemingly having no effect,
apart from an error, which can lead to confusion. However,
it must be noted that, in principle, this issue can also appear
locally, although it is more unlikely due to the smaller la-
tencies.
All in all, the prototype shows that the approach is viable,
that is, the monitoring approach originally presented in [2]
can be extended to support the remote control of test
benches without violating any of the security-related re-
quirements. The next section discusses some related ap-
proaches and solutions.
5. Related Work
The concept of monitoring and controlling test benches re-
motely is still a rather new topic in the aerospace industry
and is slowly being addressed by test equipment manufac-
turers.
There are numerous solutions on the market that make it
possible to monitor and evaluate test data remotely. But
currently, there are no solutions available that fulfill all the
requirements presented in section 2.
SystemLink4 is a solution from National Instruments for
connecting, deploying, and managing test benches online.
For the purposes of SystemLink, managing test systems
means installing software packages and rebooting devices.
This is possible via a Salt5 interface. The Salt interface can
be used to manage the IT infrastructure, even from third-
party vendors. To manage the systems via Salt, additional
software must be installed on each device. However, it is
not possible to configure or reconfigure a test bench with
different test configurations and to control the states of the
LTM or the test via control commands, like on-site. This
fails to comply with REQ15.
For remote monitoring, SystemLink uses the IoT middle-
ware AMQP. The monitored data is transmitted directly to
a server from each LTM. A server provides the transmitted
data to users via a web application. The server can be set
up on-site or in the cloud. In contrast to the presented ap-
proach, it is necessary to install additional software on each
LTM, which must then also be connected directly to the
Internet. This fails to comply with REQ7 and requires the
firewall setting to be adjusted, thus failing to comply with
REQ9. Furthermore, in order to use SystemLink in a vir-
tual and hybrid test bench (see REQ4) consisting of sys-
tems from different suppliers (see REQ3), those systems
must implement the non-standardized SystemLink inter-
face, resulting in a vendor lock-in.
Other solutions include PAK cloud6 from Müller-BBM,
Diagnostics 4.07 from Softing, and Lab Discover from
Keysight Technologies8. All of these systems claim to al-
low for remote monitoring of test benches. With some so-
lutions, it is possible to define notifications. However, it
does not seem to be possible to control the systems and in-
tervene in the tests remotely (failing to comply with
REQ15). Furthermore, it remains unclear whether they can
be used with devices from different manufacturers. This
fails to comply with REQ3.
In other industries, there are cloud-based solutions for con-
trolling industrial machines and sensors.
One approach described by H. Raju et al [6] is based on
OPC UA9 [7]. An OPC UA server collects the data from
the industrial devices and sensors (OPC UA client). To
4
National Instruments Corp., "SystemLink 18.2 Manual":
https://www.ni.com/documentation/en/system-
link/18.2/manual/manual-overview/ (Accessed May 2021)
5
Salt: "SALTSTACK Salt Documentation":
https://docs.saltproject.io/en/pdf/Salt-3003.pdf (Accessed
May 2021)
6 10
PAK cloud website: https://www.mbbm-
vas.com/en/products/data-management/pak-cloud (Ac-
cessed May 2021)
7
Diagnostics 4.0 Poster: https://automotive.soft-
ing.com/fileadmin/sof-
ETTC 2021– European Test & Telemetry Conference
transmit the data to a server in the cloud, the data must first
be translated to MQTT or another IoT middleware. Any
control commands sent to a machine via the cloud must
first be translated from an IoT middleware command to a
command that can be understood by the OPC UA client.
Using an IoT middleware usually requires all clients to be
directly connected to the Internet, which violates REQ7.
There are ways to mitigate that, such as using a gateway as
a proxy, but this would still result in a more complex solu-
tion, compared to our log message-based approach.
6. Conclusion and Outlook
This paper has presented a solution that allows users to
monitor and control test sites anywhere on the globe. It uses
the same architecture and technology stack as a cloud-
based monitoring approach that was presented recently.
Thereby, it also fulfils all the strict requirements regarding
security and cost-effectiveness. In contrast to existing ap-
proaches, the test equipment does not require a direct con-
nection to the Internet and there is no way an adversary
could gain full access to the test bench. The control aspect
has been designed to fit perfectly in the log-based nature of
the monitoring approach by encoding the control com-
mands into log messages, thereby persisting them in the log
database. The on-site gateway is the single consumer of
these control commands and could be easily deactivated in
case of a security leak, so local testing can continue.
The presented prototype proves that the approach can be
implemented in the cloud, although this is not strictly nec-
essary, and it can be deployed in a self-hosted environment
just as well, thus providing maximum flexibility.
There are still some features missing though. For example,
it is not currently possible to easily load a test bench con-
figuration. This is because the frontend has no means of
querying the available configurations. One future task
therefore is to find a suitable protocol to convey this infor-
mation.
Another open issue is the back-end API to decouple the
front end from the Elasticsearch database and allow for
more bandwidth-efficient communication.
Finally, with regard to Elasticsearch, it is necessary to eval-
uate whether the recent changes in licensing10 impact the
applicability of the approach and whether a migration to a
fork is required.
files/pdf/ae/poster/WEB_180831_1_softing_Poster_Diagn
ostics.pdf (Accessed May 2021)
8
Keysight Technologies: "Scienlab Lab Discover,"
https://www.keysight.com/de/de/assets/7018-06287/bro-
chures/5992-3238.pdf (Accessed May 2021)
9
OPC UA: https://opcfoundation.org/
FAQ on 2021 License Change for Elasticsearch:
https://www.elastic.co/pricing/faq/licensing (Accessed
May 2021)
 7. Acknowledgement
Part of this work was funded by the German Federal Min-
istry of Economic Affairs and Energy.
8. References
[1] D. H. Martinen, M. Lagalaye, J. Pfefferkorn and J.
Casteres, "Modular and Open Test Bench Architecture
for Distributed Testing," SAE International, Hartford,
Connecticut, 2017.
[2] L. Stockmann, F. Nolte and S. Reglitz, "Monitoring of
Virtual and Hybrid Test Benches," in SAE
International, Orlando, Florida, 2021.
[3] A. Himmler, L. Stockmann and D. Holler,
"Communication Infrastructure for Hybrid Test
Systems - Demands, Options, and Current
Discussions," SAE Int. J. Aerosp., vol. 9, no. 1, pp.
134-139, 2016.
[4] L. Stockmann and A. Himmler, "Monitoring and
Control of Hybrid Test Systems," SAE International,
Hartford, Connecticut, 2017.
[5] J. E. White, "RFC 707: A High-Level Framework for
Network-Based Resource Sharing," in Proceedings of
the 1976 National Computer Conference, New York,
1976.
[6] H. S. R. a. S. Shenoy, "Real-time remote monitoring
and operation of industrial devices using IoT and
cloud," in 2016 2nd International Conference on
Contemporary Computing and Informatics (IC3I),
Amity University, Noida, India, Dec. 2016.
[7] Fraunhofer IOSB-INA, "Industrie 4.0
Kommunikation mit OPC UA Leitfaden zur
Einführung in den Mittelstand," Forum Industrie 4.0,
Fraunhofer-Anwendungszentrum Industrial
Automation (IOSB-INA), Frankfurt am Main, Lemgo,
2017.
ETTC 2021– European Test & Telemetry Conference
9. Glossary
AAI: Authentication and authorization infrastructure
– provides support for authentication and au-
thorization
AMQP: Advanced Message Queuing Protocol – an open
standard for message transmission based on
middleware
API: Application Programming Interface
HTTP: Hypertext Transfer Protocol – a common appli-
cation layer protocol
HTTPS: A secure variant of HTTP using Secure Sockets
Layer (SSL) encryption
IoT: Internet of Things
LTM: Laboratory test means - a single piece of test
equipment that is part of the test environment in
a test bench
MQTT: Message Queue Telemetry Transport - a proto-
col standard for publish/subscribe-based mid-
dleware
OPC UA: Open Platform Communications Unified Archi-
tecture – an open standard, developed by the
OPC Foundation which defines a protocol for
machine-to-machine communication
UI: User interface
VHTNG: Virtual and Hybrid Testing Next Generation – a
research project that aims to create a generic and
modular architecture for an open test environ-
ment
VPN: Virtual Private Network – a network, which can-
not access from third users