i-com 2023; 22(3): 175–192
Sascha Minor*, Vix Kemanji Ketoma and Gerrit Meixner
Test automation for augmented reality
applications: a development process model and
case study
https://doi.org/10.1515/icom-2023-0029
Received October 23, 2023; accepted October 24, 2023;
published online November 9, 2023
Abstract: Poor software quality results in avoidable costs
of trillion dollars annually in the United States alone. Aug-
mented Reality (AR) applications are a relatively new soft-
ware category. Currently there are no standards to guide
the development process and testing is predominantly ad
hoc and manual. Consequently, design guidelines and soft-
ware test automation techniques are intended to remedy
the situation. Here, we present a concept for test automa-
tion of AR applications. The concept consists of two parts:
design guidelines and process model for testing AR applica-
tions, and a case study with a prototype application for test
automation. The design guidelines and the process model
are based on the state-of-the-art. The prototype application
presented in this article demonstrates test automation for a
multi-platform AR application for Android devices as well
as the HoloLens 2. The presented test automation case study
is designed to cover a large part of the functions, such as
the different interaction variants. This research work shows
that by using the proposed process model and test automa-
tion techniques, testing of some features of AR applications
can be automated. The results of this research can serve
as a basis for future research and contribution towards AR
application development standardization efforts.
Keywords: augmented reality; usability; user experience;
design guidelines; test automation
1 Introduction
Before an application is released, it must be confirmed
that it works properly. For this purpose, this software must
*Corresponding author: Sascha Minor, UniTyLab, Heilbronn University,
Heilbronn, Germany, E-mail: saschaminor@googlemail.com
Vix Kemanji Ketoma and Gerrit Meixner, UniTyLab, Heilbronn Univer-
sity, Heilbronn, Germany, E-mail: ketoma-vix.kemanji@hs-heilbronn.de
(K.V. Kemanji), gerrit.meixner@hs-heilbronn.de (G. Meixner). https://orcid
.org/0000-0001-9550-7418 (G. Meixner)
Open Access. © 2023 the author(s), published by De Gruyter.
generally be tested. Software is therefore partly verified
manually according to a prescribed test plan in a test envi-
ronment. However, manual testing is monotonous, repeti-
tive, and time-consuming [1]. As long as manual testing is
performed, the application is not developed further. Devel-
opers wait for the result of testing to fix existing bugs [2]. The
repetitive tasks of the testers lead to the fact that the testers
do not pay full attention to their work in the long run. As
a result, serious errors can be overlooked [3]. According to
the Consortium for Information & Software, the total cost of
poor software quality in the U.S. alone was over two trillion
dollars in 2020 [4]. The solution to the monotonicity and
timing problem is to automate repetitive testing [3].
Unlike traditional software, an AR application requires
additional prerequisites to be verified so that a user can
use the application without any problems [5]. The stan-
dard procedure for testing AR applications consists of a
test checklist that must be worked through manually by
testers in various situations. AR applications are generally
characterised by relatively high flexibility and surround-
ing awareness, allowing users to operate the application
location-independently and move freely in space [6].
As a result, more scenarios need to be tested. To ensure
that the quality of the software is high, automation of AR
application testing is required along with design guidelines.
Therefore, the research presented in this paper examines
the state-of-the-art in AR application design and develop-
ment with the aim to identify opportunities to maximizing
usability/user experience (UX) and automation of testing in
order to develop a blueprint for these approaches. Conse-
quently, the two research questions arise:
– What are the opportunities for increasing the usability
as well as UX of AR applications?
– To what extent can tests for AR applications be
automated?
The remainder of this article is structured as follows. In
Section 2, we present the state-of-the-art of AR application
testing. Providing a fundamental background, discussing
tools for implementing AR applications – particularly with
the Unity game engine, and related research on testing and
test automation of AR applications. In Section 3, we present
This work is licensed under the Creative Commons Attribution 4.0 International License.
176 — S. Minor et al.: AR test automation: a process model and case study
a guideline for designing AR applications which is based
on state-of-the-art and related work. Following the guide-
line, we present a process model for developing AR appli-
cations in Section 4. The process model details a guide for
implementing AR applications ensuring quality assurance.
In Section 5, we present a prototype AR application which
serves as case study for AR application test automation. We
present tools and methods for automating tests for AR appli-
cations. We also discuss the main findings in Section 5.3. We
conclude this article with a recap of the main findings and
outlook in Section 6.
2 State of the art
In this section we discuss related work and current research
and industry advancements in AR, tools for developing AR
applications and AR application testing. We begin with a
background discussion on testing and different types of
tests (Section 2.1). Then we discuss the current develop-
ment process of AR applications with a focus on tools for
testing AR applications, mainly for the Unity game engine
(Section 2.2). We conclude this section with a discussion of
related research work on test automation for AR applica-
tions (Section 2.3).
2.1 Test automation
Due to the continuous improvement of hardware capabili-
ties as well as the limited availability of AR devices, the test-
ing process in AR development mostly consists of manual
trial and error testing. For the testing process to be carried
out, the application with all required configurations must
be ported to the test device in an elaborate way [7]. From
the combination of AR and testing, new aspects, such as
dynamic, real-time, three-dimensional, multimodal interac-
tion, illumination of digital objects as well as the movement
of the user, are added that are not normally considered
when testing software.
A gross mistake in software development in general is
that software testing should not be considered as a veri-
fication step at the end of development. Due to the time
pressure at the end of a project, it could happen that testing
is neglected or reduced. However, if errors are found at
these later stages, it is more difficult to eliminate them.
Furthermore, it is possible that the same mistake will be
made again and again during development. Testing should
be seen to improve the quality of the software [8]. There-
fore, development and testing should not be considered as
separate steps but should be combined [9]. It is recom-
mended that testing be included as an iterative step in devel-
opment. If testing is rarely done because testing takes too
long, this is often a reason for running tests manually. For
this reason, testing should be automated because it can save
time and money in the long run. This means that no person
must be delegated to test all functions. One advantage of
doing this is that errors are detected earlier because tests
are performed more frequently [8].
With the help of software engineering testing tech-
niques, the manual testing effort in the development of AR
applications can be reduced. Cohn’s test automation pyra-
mid (Figure 1) [8] can be used as a basis for software engi-
neering testing techniques. The pyramid consists of unit,
service, and user interface (UI) testing parts, with complex-
ity and effort increasing upward while the number of tests
decreases [8].
2.1.1 Unit tests
Unit tests are the bottom level of the test automation pyra-
mid [8]. A unit test checks a unit of a class [10]. In most
cases, a unit is a function. These functions are to be tested
in an isolated test environment [11]. Unit tests are written
according to the AAA (Arrange, Act, Assert) pattern. In the
first section, Arrange, all required objects and variables for
the method under test are initialized. The method is called
with the specified parameters in the Act section. Finally, the
Assert section verifies that the method behaves as expected
[12]. Since unit tests display a result within seconds, this
allows errors to be identified at an early stage. Furthermore,
the tests provide rapid feedback as to whether a code change
made to the software is error-free [9].
Figure 1: Test automation pyramid (adopted from [8]).
 S. Minor et al.: AR test automation: a process model and case study
2.1.2 Service tests
The middle level of the test automation pyramid deals with
service tests. Service tests are not limited to the service-
oriented architecture, but this level refers to all functions
and services that are executed in response to an input [8].
The test types integration and Snapshot Test that can be used
as a service test are described below.
Integration testing distinguishes between two types
of tests: component tests and system tests [10]. Compo-
nent tests deal with testing of system functionalities which
interact with one another. Prior to component tests the indi-
vidual functionalities are tested at the unit test stage [9].
In contrast, system integration testing tests functionalities
across multiple systems, such as web services or database
calls [10].
Snapshot testing compares a snapshot of the user inter-
face with a reference snapshot to ensure that the user
interface remains unchanged when code changes [13]. The
snapshot is a DOM (Document Object Model) file that stores
all objects of an HTML page in a model [14]. For example, it
can be used to check whether all elements of a particular
component are present. However, the Snapshot Test does
not tell whether the elements are arranged correctly [15].
When a new element is added, the DOM file changes. For
this reason, the snapshot must be updated. The difference
between Snapshot tests and unit or integration tests is that
Snapshot tests do not confirm the correct execution of func-
tions but compare whether the user interface elements of
the application has not changed [13].
2.1.3 User interface tests
At the top level of the pyramid, the UI is tested. Usually, the
number of UI tests created is kept to the minimum possible
because UI tests are time consuming and error prone when
the user interface changes. Minimal changes to the user
interface can cause all UI tests to fail, so the tests need to be
rewritten. If it happens more often during a project that the
user interface is changed, it can lead to the UI tests not being
corrected and thus not being executed [8]. Two techniques
for performing UI tests are Visual Regression testing and
End-to-End testing.
UI test automation for AR applications is more complex
because it is more difficult to simulate user interactions with
the UI [16].
One way to test the user interface is to use Visual
Regression tests (also called screenshot tests). Visual Regres-
sion tests are used to ensure that the UI is rendered
as desired even after code changes. For this purpose,
— 177
screenshots are taken in advance (before any desired
changes or updates are implemented) as a reference from
the UI. During the Visual Regression test, new screenshots
of the UI are taken and compared with the reference images
pixel by pixel. If there are differences, these are displayed to
the tester [17].
End-to-End (E2E) tests are used to check whether the
application works as a unit. This involves testing all com-
ponents and subsystems, for example the user interface,
backend, databases and network services of the application
[9]. In this process, E2E tests should simulate real application
scenarios of potential users. Since running the tests takes
more time than all test types before, they are therefore run
less frequently. In contrast, unit tests, for example, can be
executed at the slightest code change because they take only
a few seconds. To create the End-to-End tests, tools are used
that can interact with the application’s user interface. Oth-
erwise, the user’s interaction with the application cannot be
simulated as well as the reaction to it can be verified [18].
2.1.4 Non-functional requirements
Although non-functional requirements are outside the
scope of this work, they are not negligible in software devel-
opment. For this reason, some non-functional requirements
are described as examples, since they are essential for the
acceptance of potential users. Non-functional Requirements
differ from the previous types of tests (functional require-
ments), which deal with the functional correctness of the
application [19].
If an application is too slow or inefficient, even though
it has all the desired functions, user acceptance drops enor-
mously. As a result, the application fails in the market. Per-
formance and load testing can be used to identify potential
performance problems [19].
For example, performance can be measured using a
simple method. For this the difference of the system time,
which the application needs from the start to the end of
a function, is stopped in the respective programming lan-
guage as well as computed. The meaning of performance
depends on the goal of the application. Potential prob-
lems with the performance are the visualization [19]. For
example, a high number of polygons makes digital objects
look more real. However, if the number of polygons is too
high, performance is reduced in terms of frame rate per
second [20].
Load tests are used to try to find out the maximum
limits of the application. This is important, for example, to
investigate how many users can use the application simul-
taneously so that the application works reliably under high
178 — S. Minor et al.: AR test automation: a process model and case study
load. As a result, the maximum number of concurrent users
can be limited [19].
Functionality, structure and aesthetics of an application
alone are not enough to ensure a high-quality UX. Atten-
tion must be paid to the potential users as well as to the
application context for the AR application. For this purpose,
it is necessary to conduct usability tests. Usability testing
involves observing potential users using the application
to evaluate how users interact with the application. After
observation, improvement of the application takes place to
ensure that it is intuitive and easy for users to use. Intuitive-
ness as well as usability in life-threatening situations such
as when using hospital equipment are of great importance.
Usability testing is iterative, requiring new observations of
user usage after the application has been adapted [21].
The contrast ratio of the colours of the content, for
example the text and background colour, is of enormous
importance for readability. This is especially important for
people with visual impairments. The Web Content Accessi-
bility Guidelines (WCAG) 2 of the World Wide Web Consor-
tium (W3C) are guidelines to make web content accessible
for people with a visual impairment [22]. Since the read-
ability of AR content also depends on the contrast ratio of
colours, the principle could be used to make applications
accessible to users with visual impairment.
The principle of contrast ratio cannot easily work for
AR applications because the distances to digital content can
change with the user’s movement. Therefore, changing the
font size, for example, cannot be used for low contrast.
Furthermore, this also does not consider the environment
in which the AR application is used. Nevertheless, the co-
consideration of the contrast ratio should happen. If the
environment does not change, adjusting the contrast of the
application to the environment can be done. For unknown
environments, the contrast ratio should be high. For this
reason, the use of WCAG 2 level AAA (ratio 1:7) [22] should be
done regardless of the font size of the text. Furthermore, a
combination of level AAA with Microsoft’s recommendation
[23] for readability is advantageous because the colours of
the environment are unknown. The recommendation states
that the placement of white text should happen on a dark
background. The MRTK uses a dark blue (RGB 16, 32, 106)
as the background colour. The contrast ratio of white text
(RGB 255, 255, 255) on the dark blue background is 1:14.61
[23]. By using contrast ratio for example, AR applications
become accessible for people with red-green deficiency. Peo-
ple with red-green deficiency, the most common color vision
deficiency, have difficulty distinguishing the colours red and
green [24]. Red (RGB 255, 0, 0) and green (RGB 0, 128, 0) have a
contrast ratio of 1:1.28. Thus, this combination would not be
acceptable for the application, increasing the accessibility of
the application for people with impairments.
2.2 Tools for developing and testing
augmented reality applications
AR overlays digital content on the physical world, making
the digital content appear as part of the real world in the
user’s perception. This creates a connection between the
physical and digital worlds [25]. The difference between AR
and Virtual Reality (VR) is that in AR the digital content
is rendered (transferred) into the user’s real environment.
In contrast, in VR, everything must be rendered around
the user [23]. The user is placed in a completely computer-
generated environment [25]. AR and VR are part of Mixed
Reality (MR), a blend of the real as well as the digital world.
MR enables interaction between humans, computers, and
the environment [26].
In this section we discuss tools for development, test,
and test automation of AR applications. We mainly focus
on Unity game engine compatible tools as well as libraries
that were used in this work. We select Unity as the
integration platform since it is one of the most widely
adopted tools in research and industry for implementing AR
applications [20].
2.2.1 Developing
The open-source project Mixed Reality Toolkit (MRTK)
which was initially created by Microsoft is an asset for
Unity that is suitable for the development of AR applications.
Prefabs for interaction and user interface are available in
the asset [23]. For example, MRTK includes settings for the
camera to adjust the perspective for each eye to perceive
depth [20].
Unity MARS (Mixed and Augmented Reality Studio) is
a Unity extension for creating and simulating AR content.
The extension includes a simulation mode that can be used
to test AR content in various real-world mock-ups (digi-
tal recreations of rooms and environments). This allows
upfront testing of applications in the editor before compil-
ing and running them on an AR-enabled device. For this
purpose, there are two modes: the simulation view and the
device view. The simulation view displays all content in
the simulated environment, whereas the device view is a
continuous simulation. The device view behaves as if the
device is held in the user’s hand (smartphone) or attached
to the user’s head (HMD). When the simulated device’s
camera encounters data or objects, the device scans them
and executes the designated AR content. In addition to the
 S. Minor et al.: AR test automation: a process model and case study
simulation mode, the AR application can be developed using
MARS [27].
2.2.2 Functional testing
An asset that can be used for testing is the Unity Test Frame-
work (UTF). UTF provides Unity users with the ability to
test their code in both Edit Mode and Play Mode. For this
purpose, the UTF uses an integration of the NUnit library.
NUnit is an open-source library for unit testing in .net lan-
guages. Edit mode tests are executed only in the Unity editor.
With edit mode tests, it is possible to check objects before
the scene is initialized. In contrast, play mode tests check all
objects at runtime [28].
GameDriver is an automation framework for Unity
projects. Tests can be executed both in editor mode and on
the target devices. According to GameDriver, target devices
include iOS, Android, XR, PC and consoles. With the help of
the “HierarchyPath” technology, every object in the object
hierarchy can be found and thus manipulated. Accordingly,
simulations of inputs and interactions or checks of proper-
ties are possible [29].
With Unity Automated QA, tests can be developed with-
out having to write any code. The interaction with the user
interface is to be recorded and replayed afterwards. During
the automatic replay, it is checked whether the required
user interface objects are visible and controllable. Further-
more, the replay should be executable on Android and iOS
devices [30].
AltUnity Tester is an open-source tool from Altom for
UI test automation. It offers the possibility to write tests in
C#, Python or Java and execute them within the Unity Editor
or on smartphones. The tool is available in the Asset Store.
Optionally, the AltUnity Inspector can be used. This makes
it possible to check applications outside of the Unity Editor
[31]. Furthermore, AltUnity Inspector provides the ability
to use mouse, keyboard, touch screen as well as joystick
interactions in the application for testing purposes. Via a
web socket connection, both AltUnity Tester and AltUnity
Inspector can connect and interact with applications devel-
oped in Unity [32].
Airtest by NetEase Games is a cross-platform UI tool for
automated testing. It works using image identification. The
supported platforms for testing with Airtest are Windows,
Android and iOS. To test Unity applications with Airtest, the
“Poco SDK” is required for UI element detection. Besides
Unity, the SDK can also be used for native Android and iOS
development [33].
— 179
2.2.3 Non-functional testing
Although testing non-functional requirements is not the
main focus of this work, tools are still listed for the sake
of completeness to show how non-functional requirements
can be tested.
The asset Performance Testing Extension for Unity Test
Framework extends UFT with performance testing capabil-
ities. This makes it possible to perform various measure-
ments on the application. Afterwards, a performance test
report shows a detailed analysis of the measurements [28].
Another asset for performance testing is Project Audi-
tor. Project Auditor is an experimental analysis tool that ana-
lyzes assets, settings, and scripts of a Unity project and sug-
gests recommendations for improvement. An automatically
generated report consists of code and settings diagnostics,
build report and asset information. The code and settings
diagnostics shows a list of potential issues that can affect
performance as well as memory. From the information, rec-
ommendations for the problems are obtained to resolve the
issues. The build report shows information about the time
and size of the last build to be able to improve it [34].
For the calculation of the contrast ratio WebAIM offers
an online calculator. It also shows the background as well
as the text color and whether the contrast ratio would be
passed according to WCAG AAA [22].
2.3 Towards automated augmented reality
tests
In the previous sections we have discussed different types
of software tests, we have also looked at the tools for devel-
oping and testing AR applications. However, there remains
several challenges for testing AR applications. There has
been much effort in research studies on techniques to
approach these challenges for testing AR applications. In
this section of the state-of-the-art we want to discuss some of
the research effort towards AR tests and automation of these
tests. We discuss related XR (AR, VR, MR) research on the
topic since the results from XR domain can be transferred to
another, followed by some specific research on AR test and
test automation.
Most of the research on XR test and test automation
has been focused usability, UX, and other non-functional
requirements [35–37]. Ansari [35] proposes an tool for auto-
mated user assessment XR experiences that includes both
objective and subjective measures. The author identifies the
gap in in-depth evaluation of UXs and thus propose a tool
180 — S. Minor et al.: AR test automation: a process model and case study
providing standardized, reliable and quick assessment of
UXs. Figueira and Gil [38] in their research work do not focus
on non-functional requirements, but rather on functional
requirements; proposing a tool “Youkai” for unit testing of
VR/AR applications. The tool uses python scripts running
in a docker instance to execute the unit tests, which are
capable of moving the virtual camera and finding objects
for test. The tool is evaluated in a case study and found
to facilitate the process of unit testing, reducing developer
effort and more flexible than comparable tools. However,
the authors only present a limited scenarios where the
tool is useful; mainly for UI elements testing, image/texture
loading and scrolling. Corrêa et al. [39] propose a tool
“VR-ReST” to automatically generate test requirements and
data for VR applications. While the authors evaluate the tool
with a game based on a proprietory framework, the tool
is relatively dated and it is not clear how the tool can be
integrated with other common XR development tools such
as game engines. Bierbaum et al. [16] describe the testing
of VR applications in their paper. In object-oriented designs,
traditional unit tests test methods on various inputs to verify
that the method executes correctly. At a higher level, instead
of testing individual methods, common components can be
tested in integration tests. Because of interaction, the differ-
ence in testing VR applications is in UI tests, as interaction is
fundamentally different from mouse input. To this purpose,
Bierbaum et al. used recorded interactions, which allowed
the UI test scenario to be executed [16].
For AR specifically, Börsting et al. [40] define a research
agenda for AR application development and identify testing
as one of the main components of AR development. The
authors identify the different types of tests, and challenges
of AR application testing. In their findings and research
agenda they acknowledge that while tools and methods
have been developed for traditional tests, these still need
to be translated to AR applications. The authors argue that
automate testing of AR applications it is necessary to model
and formalize new interaction modalities and incorporate
elements of the physical environment. Porfirio et al. [41]
also focus their research on automated testing of AR appli-
cations. The authors present a Model Based Testing tech-
niques to generate executable test scripts from Finite State
Machines. The authors demonstrate that the solution is
feasibility for UI tests with two AR applications. The tool
ARCHIE [42] presented by Lehman et al. in their research
helps support the process of AR application testing. While
the tool does not offer test automation for AR applications,
it automatically logs relevant data when the AR applica-
tion is tested in situ; using profiles to switch between con-
figurations. The authors evaluate the processing overhead
on incorporating the tool in existing AR applications and
find that in introduces a maximum 12 % additional pro-
cessing overhead and 2 % for profile switching. While this
tool assists in collecting data during tests, the test itself is
conducted manually by testers.
The related research work we have discussed for XR in
general and specifically for AR applications mostly focus on
UI tests. Some of the solutions are also focused on generation
of test case requirements only, while others do not directly
integrate with game engines. It is necessary to research
systems for test automation of not only UI elements but also
3D objects in the AR application and interaction. Since game
engines such as Unity Engine and Unreal Game Engine are
also very common in development of AR applications, it is
also necessary to research techniques to directly integrate
test automation with this familiar development workflow
and tools.
3 Augmented reality design
guidelines
In Section 2, we have discussed AR application testing and
the different types of tests for improving the quality of AR
applications. However, the quality of AR applications does
not only depend on testing. Since different AR devices must
be considered in both the development and evaluation of AR
applications, design principles play a major role in ensuring
high quality, improved usability and UX.
According to the Rule of Ten, the cost of fixing bugs
increases by a factor of 10 with each stage of development,
so it is beneficial to fix or avoid bugs as early as possible in
software development [43].
For effective as well as efficient human-machine inter-
action, the design must be developed based on user capa-
bilities, experience, and expectations. In AR development,
the design is described as a great challenge, which is why
the topic has a high relevance [44]. It must be considered
that both beginners and experienced users can use the
application [45]. Therefore, in this section, we enumerate
important design guidelines based on current research and
industry standards, as each hardware developer has its own
design guidelines and standards. These design guidelines
are intended for the design process of AR applications, espe-
cially when developing cross-platform AR applications.
3.1 Providing information
When using AR, it is possible that a lot of information is
needed. For this reason, it is important to use and place it
correctly.
 S. Minor et al.: AR test automation: a process model and case study
The user should be aware of the reason why the appli-
cation requires certain permissions [46].
– Therefore, the relevance and usefulness of the permis-
sions for the application should be made clear to the
user (e.g., access to camera is needed for AR) [46].
– Instructions and information should not include techni-
cal terms and AR terminology, as these are intimidating
to inexperienced users. Instead, colloquial terms should
be used [47].
It should be clear to the user how much physical space in
their immediate real-world surrounding is required to use
the AR application. Nevertheless, different environmental
conditions should be considered, such as available space
and real obstacles, because the user’s movements can result
in dangerous outcomes if they collide with obstacles [46].
The digital content must be adapted to the environ-
ment. If a digital object is to be placed on a surface, the object
must be adapted to the size of the surface [46].
Some AR applications try to recognize surfaces in the
environment as planes using feature points and display
them [46].
– In this case, the user should be shown when the appli-
cation is scanning the environment to detect surfaces
[47].
– Digital objects should not be placed exactly at the edges
of the detected surfaces because the surface boundaries
are approximations. The values could change as the
environment is further analysed [47].
3.2 User interface
In addition to the information, the UI must also be placed
correctly so that the immersion does not suffer.
– It should be avoided that the screen of smartphones
and tablets is overloaded with controls and informa-
tion so that the AR experience (immersion) does not
suffer [47].
– If a control element must be permanently available, the
control element should be present as a two-dimensional
element at the edge of the screen [47].
– Only information that is needed should be displayed
[47].
– A visual indicator shall be available to show users more
information when needed. By tapping on it, more infor-
mation can be displayed [47].
For HMDs, such as Microsoft’s HoloLens 2, the user interface
must not obscure the user’s view. The UI elements could lead
— 181
to a safety hazard if the user cannot see the real environ-
ment [48].
– The UI must not be too small, because interaction with
small elements is difficult [48].
– The UI does not always have to be displayed but can be
accessed using gestures [48].
– If a user has problems with interaction by gestures due
to an impairment, alternative operating options would
be beneficial [48].
3.3 Interaction
A new technology also brings new interaction possibilities.
Therefore, if a familiar standard is available for a particular
action, which can be appropriately implemented in AR, it
should be used. This way, users do not have to learn a
completely new method [46].
– Interactions with digital objects should preferably be
enabled by direct touch since direct touch appears
more intuitive for users. This is comparable to interac-
tion with smart devices [47].
– Standard and familiar gestures should be enabled [47].
– For tasks that require a user to move, textual informa-
tion may not be sufficient in some cases. Therefore, it
can be helpful to display the exact movements includ-
ing speed to the user [49].
– When translating (moving) objects, limits should be
added so that the user can see the maximum distance he
is allowed to move an object. This can prevent objects
from being out of the field of view (FOV) so that they
can be manipulated [46].
– The manipulation of objects, such as moving, rotat-
ing, or scaling, should work with the usual interac-
tion options (particularly on smart devices). This means
that objects can be manipulated with standard touch
gestures [46].
For HMDs, head and neck movements must be considered.
Horizontal head movements are more appropriate than ver-
tical head movements since vertical movements are more
strenuous for the neck as well as the eyes [23].
– Interactions that put the user in unnatural head posi-
tions should be avoided [23].
The gestures must be described in a documentation. For
each gesture, there should be a name, a visual description
in the form of a drawing, a photograph, or animation; for
example, a textual description of how the gesture is to be
performed [50].
182 — S. Minor et al.: AR test automation: a process model and case study
3.4 Digital content
This section addresses how content can be used to increase
immersion with AR content.
The real world should merge with the AR content.
– To this end, shadows, illumination, occlusion (masking
of real objects), reflection, and collision must be used,
giving AR objects a natural place in the real world [46].
– For the digital content to appear stable, it should
be rendered at the maximum available render rate,
also referred to as frames per second (FPS), of the
device [23].
Full-size objects are better for immersion of the user. Here,
it is important to note that AR objects can be viewed three-
dimensionally from all sides [46].
– For this reason, all surfaces of the objects must con-
tain textures. Textures make objects appear more
realistic [46].
– One factor that affects the performance of the render
rate on HoloLens is the number of polygons in an AR
object. The higher the number of polygons, the more
real the objects appear. In contrast, too high polygons
count can decrease the FPS [20].
– It is good practice to avoid textures that are too large,
because they cannot be loaded and rendered as quickly
[49].
– Lighting can lead to depth perception problems if, for
example, shadows are missing. As a result, digital con-
tent will not appear correctly placed [49].
– Also, relevant to correct placement are the size, occlu-
sion, and texture of the content [49].
The optimal distance for placement of digital content from
the current position of the user is between 1.25 m and 5 m. If
the content is too close, the eyes try to avoid double images,
which can lead to faster eye fatigue [23].
– Digital content should never be displayed closer than
40 cm because of eye focus effort [23].
3.5 Readability
An important point for the UX of AR is readability. Read-
ability refers to the interpretation and recognition of text,
symbols, and characters [51].
Factors that can affect readability include font, font
size, spacing, lighting, and colours of text, symbols, char-
acters, and background colour. Too much or too little illu-
mination can cause readability problems [51]. In addition,
the readability of the text depends on the display properties
(pixel density, brightness, and contrast), lens properties,
and font properties (spacing, font type, font/background
colour) [23].
– Although digital content in AR is used in three dimen-
sions, text should be used in two dimensions because
two-dimensional text is easier to read.
– Fonts used must have sufficient stroke width. Microsoft
uses the “Segoe UI” font for its mixed reality content. In
addition, “Arial” and “Helvetica” are recommended by
Microsoft [23].
– Since the user’s environment may have a complex phys-
ical background that makes white text difficult to read,
the text should be supported by a digital dark back-
ground. Dark colours are transparent in additive colour
systems [23].
3.6 Accessibility
Ensuring accessibility impacts how many people can use an
application. The more the design and development focuses
on accessibility, the more people can benefit from the appli-
cation. Additional integrated features can improve accessi-
bility for people with special needs [52]. People with impair-
ments make up about 20 % of the population. As people
age, functional limitations increase steadily. Accessibility
through additional integrated features benefits not only
aging people and people with impairments, but a much
broader range of people [53]. Microsoft estimates that 57 %
of computer users benefit from the use of accessible tech-
nologies, such as zoom [54].
– If a user is unable to move, there must be alternative
ways to be able to use the application in favour of
accessibility. For example, a user should be allowed to
move and rotate an object if they are unable to move
around the object [46].
– If the user’s movement is even further restricted, voice
control should be enabled to assist [52].
– AR application that output relevant information via
audio or video must provide an alternative in the form
of subtitles. This must have sufficient contrast ratios.
Furthermore, the subtitle should be placed in such a
way that other content is not obscured [52].
3.7 Problems encountered during the use of
AR
When using a new technology, many problems can arise,
among others, with beginners. The following are solutions
to avoid them.
 S. Minor et al.: AR test automation: a process model and case study
Users may forget the environment around them while
using AR.
– Therefore, users must be made aware that there may be
risks such as objects or people around them [46].
– The application should be designed in a way that dis-
courages or avoids users actively moving backwards
when performing actions, as the risk of collision is
much higher when moving backwards [46].
– If a large AR object is placed too close to the user, it
could cause the user to take a few steps backwards
since the user cannot see the entire object [46]. This is
because the FOV enabled by the device is significantly
smaller than the user’s FOV [55]. AR applications using
VR HMDs with pass-through video (such as Apple Vision
Pro1 and Meta Quest Pro2 ) have a significantly wider
FOV. However, for some application scenarios it is still
desirable to use AR devices. For example, critical AR
applications requiring high reliability such as military
or health care AR applications. In such applications, loss
of power or other failure in the VR HMD would impede
the user’s view. For such critical applications which still
require AR devices, the relatively small FOV of current
AR devices still presents a limitation.
There may be limitations in detecting surfaces if surfaces do
not have textures, the environment is under- or over-lit, sur-
faces are transparent or reflective, and there is movement
of surfaces. The more people and real-world objects present,
the more difficult tracking as well as occlusion is for the AR
device [46].
– The user must be informed what the problem is and
how to fix it if there are limitations [46].
Using bright colours on a large area in holographic displays
can cause faster eye fatigue for the user, as digital content
with bright colours requires more light [23].
– The use of a dark colour scheme is recommended to
minimize the problem of eye fatigue [23].
If digital content is either not displayed or rendered in the
wrong place, it may be due to poor tracking or other errors
resulting from the sensors of the headset or the camera of
the smart device. The sensors, as well as the camera, could
have issues with lighting or the environment not having
enough texture. Such errors could also result from the sen-
sors been covered by the user’s hair or hands [23].
1 https://www.apple.com/apple-vision-pro/.
2 https://www.meta.com/quest/quest-pro/.
— 183
Considering the case of AR applications deployed on
Android devices, if there is no camera image available on an
Android device, it is possible that the user has not granted
camera access permissions for all apps. For this reason, the
user must be made aware that on Android 12 (API level 31)
or higher, the camera can be blocked. The “Block camera”
setting must be set to “Off” in the system settings [46].
4 Process model for test
automation of AR applications
In this section we present a process model for testing AR
applications. This process model serves as a guideline for
quality assurance in the development process of AR appli-
cations. It incorporates both functional and non-functional
tests for quality assurance. We use this process model for
the implementation of the case study in Section 5; however
for the case study we focus only on the functional tests.
Since neither a concept nor a standard for test automa-
tion of AR application is known, many companies skip the
step of testing in software development. The applications
are exported to the AR devices in a time-consuming manner
and tested according to the “trial and error” method. During
this, design mistakes are also made, as there is no uniform
AR design standard, but each hardware manufacturer sets
up its own design principles. In addition, quality character-
istics are not inspected at the end of development. For this
reason, a concept in the form of a process model for test-
ing AR applications was developed based on the previous
results.
A process model describes methods that can be used in
the implementation of projects of the same type. Most pro-
cess models are divided into phases. The aim is to develop
the process model as a measure for improving the quality
of AR applications. This makes it possible to define a stan-
dard to assist in the communication of all parties involved
in the development [56]. However, not all phases can be
automated in this process model for testing AR applications.
The automation of all tests, especially the tests for checking
the quality features, is also not possible in other types of
software projects.
Even if the Design belongs to the conception as well
as the design phase of a software project, it is important
to include the design in the process model. The design is
essential for the usability of the application and thus for
the acceptance of the potential users, which is checked in
the last phase. Due to the different AR devices as well as
the different experience levels of the users, the implementa-
tion of design for AR applications is associated with special
184 — S. Minor et al.: AR test automation: a process model and case study
challenges. Important issues include preparation, UI, inter-
action, digital content, accessibility, and potential problems
that may arise. As each hardware developer provides their
own design suggestions, the design principles developed
in this paper should be incorporated to assist. The earlier
errors can be avoided, the less effort (time and money) is
required to improve the quality of the application.
Unit tests can be automated for the most part, along
with service testing as well as UI testing. With unit tests, it
is possible to test functions in two ways. Verification tests
are used to check if the functions work as intended with
certain values. Whereas falsification tests are used to check
for incorrect inputs so that unexpected inputs are caught
and do not cause the application to crash. Unit tests provide
the fastest feedback on whether a code change is causing a
problem. The AAA pattern is widely adopted for the imple-
mentation of unit tests. Furthermore, this pattern can also
be adopted for the other types of tests. Unit tests should be
created at the latest after completion of a feature.
Service tests are used to check related functions and
services. Integration tests can be used for this purpose. An
integration test should be implemented for each new related
feature. Furthermore, Snapshot tests can be used to check
that the user interface has not changed due to code changes.
A reference list is compared with a current list of the object
hierarchy to ensure that all elements are present. With the
Snapshot Test, no statement can be made about a correct
arrangement. For this purpose, the UI tests are needed.
User interface tests are the most elaborate, yet they
should not be neglected. On the one hand, Visual Regression
tests compare a reference image with a current screenshot
pixel by pixel to check whether all elements are arranged
correctly. On the other hand, E2E tests can be used to
check the entire application under real application scenar-
ios. Here, tools are used that support the creation of E2E
tests.
Figure 2: Process model for test automation of AR applications.
Quality features of an application are the non-
functional requirements, which is why the test types differ
from the ones before. The quality features are important
for user acceptance. Examples of quality features are acces-
sibility, efficiency as well as usability. One way to increase
accessibility is with contrast ratio, which benefits people
with red-green deficiency. Efficiency can be assessed with
performance and load tests. The most important quality
feature is usability, which is more difficult to assess due to
different AR devices as well as different experience levels of
users. Therefore, potential users should be involved in the
development process of AR applications from the beginning
and observed while using the application to enable an intu-
itive application.
A summary of the process model is shown in Figure 2.
5 AR application test automation: a
case study
5.1 Developing the prototype augmented
reality application
Considering that AR applications run on smartphones
and tablets as well as HMDs, a cross-platform applica-
tion is developed that works on both Android devices and
HoloLens 2. This will allow testing of different devices, high-
lighting manufacturer differences. Especially regarding the
different interactions for controlling the application (ges-
ture, touch and voice control), it can be assessed whether
a uniform test concept can be implemented by having all
possible control variants. For this goal, digital objects as well
as user interface elements will be integrated into the AR
application. The digital objects will be used to implement
 S. Minor et al.: AR test automation: a process model and case study
the functions rotation, translation and scaling by touch and
gesture control. Buttons will be used to represent the inter-
action with the user interface. Thus, not only the interaction
variants touch and gesture control should be possible, but
also the barrier-free option of voice control. One of the
three button actuation variants will change the colour of the
digital object or replace it with another object, among other
things. Users should be able to connect to each other via the
Internet. Consequently, a user can see when another user
has changed his digital object. Hence, it is possible to check
how different devices react to the changes of the digital
objects.
The development environment used was Unity. The
MRTK enables cross-platform development for prototype
AR application. However, using MRTK for Android and iOS
is experimental, as not all features are available for Android
and iOS. Furthermore, PUN was used for the connection
between users, as PUN also supports cross-platform devel-
opment. With the guidance from Microsoft [57] on how to
deploy MRTK for Android and iOS, it was not possible to
develop an application. Some adjustments had to be made
to use MRTK for Android, which should also run on iOS with
minor adjustments.
A prototype AR multiplatform application called
“Compañero Cube” (Figure 3) was developed as the basis
for test automation. The idea behind this application is that
users connect through the application. After connecting to
the application, an AR cube is placed in front of the user.
More cubes from other users can be seen if more users are
connected. The name of the user is displayed under each
cube. The cube can be moved, rotated as well as scaled.
Furthermore, buttons are available under the cube to
interact with the other users:
Figure 3: A user indicates in the Compañero Cube application that he
wants a coffee break.
— 185
– “Coffee Time” changes the cube to a coffee cup symbol-
izing that the user is available for a coffee break
– “I’m Hungry” changes the colour of the cube to orange.
This allows the user to represent that they are hungry
and ready for a lunch break
– “Party!” shows that the user has accomplished some-
thing special. The cube switches between the colours
red and white.
– “Neutral” resets the cube to the original shape and
position.
Compañero Cube is compatible with Android devices run-
ning Android 7 or higher (API level 24 or higher) and
HoloLens 2.
5.2 Testing the prototype augmented reality
application
To be able to write and execute Unit, Integration, Snapshot
and Visual Regression tests in Unity, the UTF was used as a
basis. The tests in the Test Script can be structured according
to the AAA pattern. If the initialized variables and objects
of the Arrange section are needed frequently, they can be
initialized using the Setup method. The method must be
tagged “[SetUp]” so that the method is called before the tests.
The Act and Assert sections are called in the test method. The
test method must be tagged “[Test]”. In some cases, the setup
should be cleaned up after the tests, for example, because
changes were made to the application in the setup method.
For this the method “TearDown” should be provided with
the tag “[TearDown]” and invoked as last the step [58].
The UTF also provides “UnitySetUp”, “UnityTest” as
well as “UnityTearDown”, which must be provided with the
respective tag. The methods are identical to the standard
methods, except that they allow yielding statements [58].
Yielding statements can, for example, pause statements and
resume them after a specified time (a frame, several sec-
onds, changing a scene, etc.) [59].
In Unity, each script inherits from the base class
MonoBehaviour [59]. The problem with inheritance is that
no instantiation of MonoBehaviour with the new operator
is possible. This makes it difficult to isolate an entity to be
tested. For this problem, the Humble Object Pattern can be
used. For this purpose, the logic of the script that does not
use the Unity API should be extracted into a separate class
and referenced to that class. When the testable entity needs
information about the logic, it delegates it to the extracted
class [60].
186 — S. Minor et al.: AR test automation: a process model and case study
5.2.1 Unit tests
Unit Tests are used to test individual units. In this regard,
there should be as few elements in the test as necessary.
For example (Figure 4a), to test the functions of the AR
application, a cube was created. The script with the func-
tion to be tested is added to the cube (Arrange). The func-
tion is executed, changing the colour of the cube to black
(Act). Finally, the function is checked to see if it works as
intended. In this case it means checking if the cube is black
(Assert).
5.2.2 Integration tests
In the Integration Tests, units that interact with each other
are to be tested. For this purpose, the scene with the
required elements can be loaded asynchronously in the
Arrange step so that all interacting elements are present.
In the process, it may happen that elements not required
for the tests (camera, light, AR components, etc.) are loaded.
The more elements there are in a scene, the more computing
capacity is used, which decreases the speed of the tests.
An alternative for loading a scene including unnecessary
elements can be prefabs. The required elements could be
stored in one or more prefabs. In the arrange step, the
Figure 4: Unit and integration Tests.
required prefabs can be loaded. Fewer elements are loaded,
which is why the test runs through faster than when loading
a scene. After that, the Act and Assert steps are to be per-
formed as in the case for the unit tests.
Figure 4b shows an example Integration Test. A scene is
loaded in which a button and a cube are located (Arrange).
The button is executed, replacing the cube with a coffee cup
(Act). Subsequently, it is checked whether the coffee cup is
present (Assert).
With the help of the “Test Utilities” package from the
MRTK asset, hand movements can be simulated. As a result,
the various interaction options, such as rotation, scaling
and translation, as well as activation of buttons by touch
and gesture control can be tested in a simulated manner.
Furthermore, the simulation of voice input for activating
buttons, for example, is possible.
5.2.3 Snapshot tests
Snapshot tests compares a reference file to the current state
to verify that all elements are present. Here, a reference text
file containing the elements of the scene is to be compared
with the current text file of the scene elements. For this
purpose, a script was implemented which stores in a text
file the following properties of each element in the scene:
 S. Minor et al.: AR test automation: a process model and case study
Figure 5: JSON schema of recorded properties for each scene element.
the name, position, rotation, scale, materials, attached script
components, and its active state (Figure 5).
Snapshot tests can be run in UTF as Edit Mode tests,
since no elements are required at runtime. The text files are
read in via StreamReader and converted to strings using the
ReadToEnd method. Thereby the strings are compared with
each other. At the end the StreamReader objects should be
closed, so that the assigned resources are released again.
If the strings are different, the text files can be com-
pared in Python. The “difflib” library is used for this pur-
pose. difflib compares sequences to find partial sequences
Figure 6: HTML file shows differences [61].
— 187
or differences. With “difflib.HtmlDiff” differences between
two strings in a HTML format can be generated as a result
(Figure 6) [61].
The left side of the HTML file displays the text of the
reference file (reference content of the scene hierarchy) and
the right side displays the current text (current content of
the scene hierarchy). Text marked in red (left) means that
the text of the reference file is not present in the current
file. Instead, text highlighted in green (right) represents the
text added in the current file. In contrast, text highlighted in
yellow (left and right) indicates small changes.
Simplified, the process of Snapshot Test is shown in
Figure 7.
5.2.4 Visual Regression
Visual Regression tests compare a screenshot with a refer-
ence image. The reference image and the screenshot must
be loaded as textures. The hash values of both textures are
calculated and compared. The hash value is a 128-bit num-
ber that returns different values for the smallest differences.
If the hash values are identical, so are the images.
However, if the hash values differ, the methods of a
Python script are used to calculate and show differences
using “OpenCV”. OpenCV is an open-source software library
for image processing, computer vision and machine learn-
ing [62].
Figure 8 shows the reference image of the cube (top left)
and the current screenshot (top right) as well as the images
calculated with OpenCV of the deviating area (bottom left)
and the deviating part marked with a red box (bottom right).
The deviating area is calculated by subtracting the pix-
els of the current screenshot from the pixels of the reference
image. The difference is saved as a new image [63]. Alter-
natively, for the deviating part marked with a red box, the
absolute difference between the images is calculated first.
Then, the absolute difference is converted to gray-scale,
188 — S. Minor et al.: AR test automation: a process model and case study
Figure 7: Snapshot test process.
Figure 8: Reference image, current screenshot, deviating area and
deviating part marked with a red box.
which allows binarization of the difference (pixel values
below three become black, all values above become white).
With the help of the binarization, the changed contours can
be found, which are marked in red [64]. Figure 9 shows the
summarized Visual Regression process.
5.2.5 End-to-End tests
Whether the application works as a whole is checked using
E2E test tools. All components and subsystems are to be
tested in real application scenarios. There are problems
with this, especially with support for MRTK and HoloLens 2.
In general, most assets and tools promise AR compatibility.
Compatibility is only given when developing with Unity’s UI
system. Once MRTK including the customized UI for AR is
used, the testing capabilities do not work.
In addition, none of the E2E tools provide testing capa-
bilities for motion and changing background colors and
lighting conditions.
5.2.6 Performance tests
Since both using MRTK for cross-platform development
and Project Auditor are experimental, the result of the
performance test has to be questioned in some places.
For example, code issues and unnecessary files were dis-
played, which are needed for Android devices or the
HoloLens. Nevertheless, Project Auditor can suggest opti-
mization improvements when developing for one device.
5.3 Evaluation of AR application test
automation
The results of this case study show that automating tests
for AR applications is possible to a certain extent. Using
the prototype application presented in this work, it was
possible to show that Unit and Integration Tests work for
HoloLens and Android devices. For the Integration Tests, it
is particularly important that all interaction variants can be
tested, which makes the development of a cross-platform
application useful so that it is not necessary to develop a
separate application or maintain a separate codebase for
each operating system. In addition, this can ensure part of
the accessibility of the application, as the alternative inter-
actions can be tested.
Given the Snapshot tests, a script was developed that
saves all elements of the scene hierarchy in a text file.
However, in Unity, this type of test is not strictly necessary
for rendering since all elements are displayed in the Scene
window. Nevertheless, if rendering does not work on the
target device, the problem is elsewhere. Regardless, the
Snapshot Test has its uses for automating AR applications.
Small changes to an element, such as trying out a setting,
will show up if they have not been undone.
 S. Minor et al.: AR test automation: a process model and case study
Figure 9: Visual Regression test process.
Visual Regression tests are used to compare the appear-
ance of the AR application. This type of test works in the
Unity Editor, using two comparison algorithms for the dif-
ferences in pixels. To check the appearance of 3D elements,
several images from different angles are needed, since this
method can only compare 2D images.
In general, the providers of tools and frameworks for
E2E Tests promise AR compatibility, but this is not always
transparent. The bottom line is that the tools and frame-
works (Unity Automated QA, AltUnity Tester, Airtest and
GameDriver) only support AR applications that have been
developed based on Unity’s input system. This is not appar-
ent from the descriptions but only during the use of the
E2E tool and framework. As MRTK served as the basis for
cross-platform development in this work, the tools and
frameworks cannot recognize the interaction elements of
the application. The assets and tools would recognize the
interaction elements when developed with Unity’s UI sys-
tem, but the interaction elements would not be compatible
for HoloLens 2.
For testing the other AR aspects, such as lighting, move-
ment and room tracking, the idea was to use Unity MARS’
simulation mode to test Compañero Cube in real-world
mock-ups. In response to an inquiry about compatibility
between MARS and MRTK at Unity Support, it was stated
that there is no compatibility. However, colour and lighting
conditions can be automatically tested and adjusted using
the contrast ratio and the recommendation of a dark back-
ground with light text. Room tracking and movement must
still be tested manually on the devices.
One challenge during development and testing was the
documentation of the libraries, frameworks and tools used.
The development as well as the testing of the AR application
was made complicated by the partly missing or incomplete
documentation. Since the developers’ websites or descrip-
tions are not transparent regarding the tools’ functions, a
lot of time was unnecessarily taken up for testing.
— 189
6 Conclusions
6.1 Summary of findings and contributions
In this paper we have discussed the need for test automation
of AR applications, presented a process model, and results
of a case study for test automation of AR applications. In
this section, we recap the main findings, the contributions of
this work, discuss the limitations, and an outlook on further
research and development.
We have discussed the current state-of-the-art of test-
ing and test automation of AR applications. Given that AR
applications are more challenging to develop and present
many usability challenges, in this paper we present design
guidelines for AR applications which is a concise curation
of guidelines from state-of-the-art and other device man-
ufacturers. We presented a process model for developing
AR applications and a case study describing techniques in
implementing test automation for AR applications. Our find-
ings from the case study show that it is possible to imple-
ment test automation for AR applications. Particularly, the
use of Visual Regression tests (with image comparison) and
Snapshot tests (from auto-generated JSON data). While some
of these techniques are used in traditional web and mobile
application testing, this research shows that they would also
be effective in the context of AR application test automa-
tion. However, some aspects of AR application testing are
still challenging to automate (e.g. real-world environment
elements, and complex interaction modalities) and must be
tested manually. AR and other XR technologies are rela-
tively new, so this is an ongoing effort and there is a need
for the community to develop standards for implementing
these applications and promote test automation. All stake-
holders, including users, must be involved in the develop-
ment of standards, and standards must also be accessible to
all [65].
The process model developed in this work can serve
as a basis for norms and standards. In addition, design
190 — S. Minor et al.: AR test automation: a process model and case study
standards would be necessary as more and more manu-
facturers develop AR devices. Design principles and test
automation should become a part of efficient AR develop-
ment with the goal to increase UX which is currently one of
the main challenges to mass adoption [66].
6.2 Limitations
We have discussed a case study for AR application test
automation. However, development of AR applications con-
sists of several steps and the applications can be more com-
plicated than traditional non-immersive 2D (web/mobile)
applications. The Case Study presented in this paper con-
tributes to research and development efforts towards test
automation of some aspects of AR applications. The results
contribute towards continued development and research
to full test automation, but do not provide a holistic test
automation of AR applications. Therefore, in this section we
discuss some of the limitations of this research.
1. In this study, we have demonstrated automated testing
of only a limited set of features of AR applications. The
scale and number of features of these applications can
be considerably larger than the presented case study.
Other features such as environmental conditions and
their effects, all interaction modalities have not been
implemented for test automation.
2. All tests in this paper have been developed based on
MRTK in Unity. The lack of compatibility between MRTK
and Unity MARS implies that some of the environment
simulation features of Unity MARS could not be used
automating these environment tests. However this rep-
resents a limitation of these tools. The test automation
techniques presented in the use can be adapted for
use with other libraries or frameworks without such
limitations.
3. Due to the high number of different test types, the pre-
sented use case demonstrates techniques for automat-
ing only a subset of test types.
4. Using MRTK for Android and iOS is experimental, so not
all features of MRTK will work. For this reason, cross-
platform development is not always practical for every
project. However, with continued open source develop-
ment of MRTK, it is possible that this may change in the
future.
6.3 Future research and development
In this research work we have discussed test automation
for AR applications and demonstrated implementation tech-
niques using MRTK. This work represents a component of
the broader conversation on AR software quality. Future
advances in the underlying AR technologies, decrease in
unit production costs, and more affordable devices could
lead to increased adoption [66] and use of the technology
in several application domains. Thus the question of AR
software quality will continue to be important; therefore,
we briefly discuss future perspective and areas of continued
research and development.
There is still a high optimization potential for test
automation of AR applications. Since AR applications are
operated in a wide variety of environments that can not be
accurately predicted at development time, it is important to
develop techniques for ensuring the application can operate
accurately in the user’s environment. Tools such as Unity
MARS already provide some of these features, but there is
need for continued improvement and similar tools for other
development software such as Unreal Engine, Cryengine,
etc. It is also important that these testing tools are ade-
quately documented to facilitate adoption and reduce the
barrier to entry in test automation for quality engineers.
With increased adoption of AR applications and use
in a variety of application domains and contexts, it can be
assumed that there will be increased need for E2E tests.
Since there are many intermediary steps and a variety of
interaction modes for an E2E interaction, it would be desir-
able for all interaction variants to be simulated and for all
AR-specific aspects to be tested.
Furthermore, emulators for different devices would be
beneficial for test automation, as is standard in Android
development, for example. In this way, many devices could
be tested at once due to different display sizes, lens types or
operating systems, among other things.
Moreover, software quality could be improved by auto-
matically checking other quality features besides efficiency
and performance. It would be conceivable to check the qual-
ity feature accessibility by automating the contrast ratio. A
tool could calculate the contrast ratio during development
and display warnings if the requirements for the AAA level
are not met. This offers the potential for future research
to examine the extent to which the test automation of AR
applications can be extended to include quality features in
addition to the automation of functional tests.
References
1. Kim M. O., Coiera E., Magrabi F. Problems with health information
technology and their effects on care delivery and patient
outcomes: a systematic review. J. Am. Med. Inform. Assoc. 2017, 24,
246 − 250..
2. Baumgartner M., Klonk M., Pichler H., Seidl R., Tanczos S. Agile
Testing: Der agile Weg zur Qualität; Hanser Verlag: München, 2013..
 S. Minor et al.: AR test automation: a process model and case study
3. Vocke H. The practical test pyramid, 2018. https://martinfowler
.com/articles/practical-test-pyramid.html.
4. Krasner H. The cost of poor software quality in the US: a 2020
report, 2021. https://www.it-cisq.org/pdf/CPSQ-2020-report.pdf.
5. Colantonio J. How to do virtual augmented reality testing, 2018.
https://testguild.com/testing-virtual-augmented-reality/.
6. Makarov A. Augmented reality development guide. https://
mobidev.biz/wp-content/uploads/2020/05/augmented-reality-
development-guide.pdf.
7. Dünser A., Grasset R., Seichter H., Billinghurst M. Applying HCI
principles to AR systems design, 2007. https://www.researchgate
.net/publication/216867606_Applying_HCI_principles_to_AR_
systems_design.
8. Cohn M. Succeeding with Agile: Software Development Using Scrum.
The Addison-Wesley Signature Series a Mike Cohn Signature Book.
Addison-Wesley: Upper Saddle River, NJ and Munich,
2010.
9. Whittaker J. A., Arbon J., Carollo J. How Google Tests Software: Help
me Test like Google; Life of a TE, Life of an SET, Interviews with
Googlers and More; Addison-Wesley: Upper Saddle River, NJ and
Munich, 2012.
10. Baumgartner M., Klonk M., Mastnak C., Pichler H., Seidl R., Tanczos
S. Agile Testing: The Agile Way to Quality. Springer eBook Collection,
1st ed. 2021 ed.; Springer International Publishing and Imprint
Springer: Cham, 2021.
11. Mindra D. Unit testing at the speed of light with unity test tools.
Unity Blog 2014. https://blog.unity.com/technology/unit-testing-
at-the-speed-of-light-with-unity-test-tools.
12. Microsoft. Unit test basics, 2021. https://docs.microsoft.com/en-
us/previous-versions/visualstudio/visual-studio-2015/test/unit-
test-basics?view=vs-2015&redirectedfrom=MSDN#BKMK_
Writing_your_tests.
13. Wickramarachchi V. Snapshot testing for frontends: Test whether
any characteristics of your application have changed. Bits and
Pieces 2021. https://blog.bitsrc.io/whats-snapshot-testing-and-
can-we-use-it-for-frontend-f5a441cae27a.
14. Müller P. M. Little Boxes, Teil 1: Webseiten gestalten mit
HTML CSS. Grundlagen, 2. aufl. ed.; Addison-Wesley Verlag, s.l.,
2009.
15. Hartmann N., Zeigermann O. React: Grundlagen, fortgeschrittene
Techniken und Praxistipps − mit TypeScript und Redux, 2.
überarbeitete und erweiterte auflage ed.; dpunkt.verlag:
Heidelberg, 2020. http://www.content-select.com/index.php?id=
bib_view&ean=9783960884194.
16. Bierbaum A., Hartling P., Cruz-Neira C. Automated testing of
virtual reality application interfaces. In Proceedings of the Workshop
on Virtual Environments 2003 − EGVE ’03; Kunz A., Deisinger J., Eds.;
ACM Press: New York, New York, USA, 2003; pp. 107 − 114.
17. Bazurto Gómez N., Guerra Gómez J. A., Linares Vásquez M., Alvarez
Martíńez D. Visual regression testing for information
visualizations, 2018. http://hdl.handle.net/1992/34290.
18. Da Costa L. Testing JavaScript Applications, 1st ed.; Manning
Publications and Safari: Erscheinungsort nicht ermittelbar and
Boston, MA, 2021..
19. Kleuker S. Performance- und lasttests. In Qualitätssicherung durch
Softwaretests; Kleuker S., Ed.; Springer Fachmedien Wiesbaden:
Wiesbaden, 2019; pp. 355 − 395.
20. Ong S., Siddaraju V. K. Beginning Windows Mixed Reality
Programming; Apress: Berkeley, CA, 2021.
— 191
21. Stiles-Shields C., Montague E. Usability testing. In Encyclopedia of
Behavioral Medicine; Gellman M. D., Ed.; Springer International
Publishing: Cham, 2020; pp. 2294 − 2296.
22. WebAIM. Contrast and color accessibility: Understanding wcag 2
contrast and color requirements, 2022. https://webaim.org/
articles/contrast/.
23. Microsoft. Start designing and prototyping − mixed reality, 2021.
https://docs.microsoft.com/en-us/windows/mixed-reality/design/
design.
24. Bhakta P. Red Green Deficiency: Causes and Consequences;
Universidad Interamericana de Puerto Rico: Escuela de
Optometría, 2017. https://books.google.de/books?id=Ndp_
zQEACAAJ.
25. Schmalstieg D., Höllerer T. Augmented Reality: Principles and
Practice; Addison-Wesley usability and HCI series, Addison-Wesley:
Boston and Columbus and Indianapolis, 2016.
26. Wen Q., Jahiu D., Sherer T., Tieto V., Ferrone H., Frazier K., Coulter
D. What is mixed reality? 2022. https://docs.microsoft.com/en-us/
windows/mixed-reality/discover/mixed-reality.
27. West T. Labs spotlight: unity mars. Unity Blog 2019. https://blog
.unity.com/technology/labs-spotlight-project-mars.
28. Technologies U. Performance testing extension for unity test
framework, 2021. https://docs.unity3d.com/Packages/com.unity
.test-framework.performance@2.8/manual/index.html.
29. GameDriver. Gamedriver, 2022. https://www.gamedriver.io/.
30. Technologies U. Automated qa, 2022. https://unity.com/de/
products/automated-qa.
31. Altom. Altunity tester − ui test automation, 2022. https://
assetstore.unity.com/packages/tools/utilities/altunity-tester-ui-
test-automation-112101#description.
32. Consulting A. Altunity tester documentation, 2022. https://altom
.gitlab.io/altunity/altunitytester/tester.html.
33. NetEase. Airtest project docs, 2022. https://airtest.doc.io.netease
.com/en/.
34. Technologies U. Projectauditor, 2022. https://github.com/Unity-
Technologies/ProjectAuditor.
35. Ansari S. G. Toward automated assessment of user experience in
extended reality. 2020 IEEE 13th International Conference on
Software Testing, Validation and Verification (ICST) 2020, 430 − 432;
https://doi.org/10.1109/ICST46399.2020.00056.
36. Harms P. Automated usability evaluation of virtual reality
applications. ACM Trans. Comput.-Hum. Interact. 2019, 26, 1 − 36;.
37. Vi S., da Silva T. S., Maurer F. User Experience Guidelines for
Designing HMD Extended Reality Applications; Springer International
Publishing: Cham, 2019; pp. 319 − 341.
38. Figueira T., Gil A. Youkai: A Cross-Platform Framework for Testing
VR/AR Apps; Springer Nature: Switzerland, 2022; pp. 3 − 12.
39. Souza A. C. C., Nunes F. L. S., Delamaro M. E. An automated
functional testing approach for virtual reality applications. Softw.
Test. Verific. Reliab. 2018, 28, e1690;.
40. Börsting I., Heikamp M., Hesenius M., Koop W., Gruhn V. Software
engineering for augmented reality − a research agenda.
Proc. ACM Hum.-Comput. Interact. 2022, 6, 1 − 34;.
41. Tramontana P., Luca M. D., Fasolino A. R. An approach for model
based testing of augmented reality applications, 2022.
42. Lehman S. M., Ling H., Tan C. C. Archie: a user-focused framework
for testing augmented reality applications in the wild. 2020 IEEE
Conference on Virtual Reality and 3D User Interfaces (VR) 2020,
903 − 912; https://doi.org/10.1109/VR46266.2020.00013.
192 — S. Minor et al.: AR test automation: a process model and case study
43. Mellis W. Projektmanagement der SW-Entwicklung: Eine umfassende
und fundierte Einführung; Springer eBook Collection Computer
Science and Engineering, Vieweg+Teubner Verlag: Wiesbaden
and s.l., 2004.
44. Börsting I., Heikamp M., Hesenius M., Koop W., Gruhn V. Software
engineering for augmented reality − a research agenda. Proc.
ACM Hum.-Comput. Interact. 2022, 6, 1 − 34;.
45. Dabor O., Longford E., Walker S. Design guidelines for augmented
reality user interface: a case study of simultaneous interpretation.
In 2019 11th Computer Science and Electronic Engineering (CEEC); IEEE,
2019; pp. 164 − 166.
46. Google Developers. Augmented reality design guidelines, 2021.
https://developers.google.com/ar/design?hl=en.
47. Apple Inc. Augmented reality − system capabilities − ios −
human interface guidelines − apple developer, 2021. https://
developer.apple.com/design/human-interface-guidelines/ios/
system-capabilities/augmented-reality/.
48. Davis N. Best practices for bringing AR applications to the field.
Unity Blog 2020. https://blog.unity.com/manufacturing/best-
practices-for-bringing-ar-applications-to-the-field.
49. Rolim C., Schmalstieg D., Kalkofen D., Teichrieb V. [Poster] Design
guidelines for generating augmented reality instructions. In 2015
IEEE International Symposium on Mixed and Augmented Reality; IEEE,
2015; pp. 120 − 123.
50. Deutsches Institut für Normung e. V, b. Ergonomie der
mensch-system-interaktion: Teil 210: Menschzentrierte
gestaltung interaktiver systeme. https://doi.org/10.31030/
3104744.
51. Sung Oh G., Hee Oh S. Impact of text distance and size in ar on
user experience: focusing on the implement of experimental
applications. In 2020 International Conference on Information and
Communication Technology Convergence (ICTC); IEEE, 2020;
pp. 464 − 467.
52. Deutsches Institut für Normung e. V, a. Ergonomie der
mensch-system-interaktion: Teil 171: Leitlinien für die
zugänglichkeit von software. https://doi.org/10.31030/1426650.
53. Vanderheiden G. C., Jordan J. B., Lazar J. Design for people
experiencing functional limitations. In Salvendy G., Karwowski W.,
Eds.; Handbook of Human Factors and Ergonomics; Wiley: Hoboken,
New Jersey, Vol. 90, 2021; pp. 1216 − 1248.
54. Microsoft. Accessible technology in computing: Examining
awareness, use, and future potential, 2004. http://download
.microsoft.com/download/0/1/f/01f506eb-2d1e-42a6-bc7b-
1f33d25fd40f/researchreport-phase2.doc.
55. Gong L., Fast-Berglund A., Johansson B. A framework for extended
reality system development in manufacturing. IEEE Access 2021, 9,
24796 − 24813;.
56. Brandt-Pook H., Kollmeier R. Softwareentwicklung kompakt und
verständlich: Wie Softwaresysteme entstehen; Springer Fachmedien:
Wiesbaden, 2020.
57. Microsoft. Deploying to android and ios (AR foundation)
[experimental] − mixed reality toolkit, 2022. https://docs
.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/
supported-devices/using-ar-foundation?view=mrtkunity-2021-05.
58. Technologies U. Unity test framework, 2020. https://docs.unity3d
.com/Packages/com.unity.test-framework@1.1/manual/index
.html.
59. Technologies U. Unity user manual, 2021. https://docs.unity3d
.com/Manual/index.html.
60. Paszek T. Unit testing monobehaviours. Unity Blog 2014.
https://blog.unity.com/technology/unit-testing-part-2-unit-
testing-monobehaviours#state=145691&error=consent_cancel.
61. Foundation P. S. difflib, 2022. https://docs.python.org/3/library/
difflib.html.
62. OpenCV. Opencv, 2022. https://opencv.org/.
63. Stackoverflow@nathancy. Detect and visualize differences
between two images with opencv python, 2022. https://
stackoverflow.com/questions/56183201/detect-and-visualize-
differences-between-two-images-with-opencv-python.
64. Srivastava V. Vergleichen sie zwei bilder und markieren sie
unterschiede mit python; GeekyHumans, 2021..
65. Daigl M., Glunz R. ISO 29119: Die Softwaretest-Normen verstehen und
anwenden; dpunkt.verlag: Heidelberg, 2016.
66. Coie P. 2020 augmented and virtual reality survey report, 2020.
https://www.perkinscoie.com/images/content/2/3/231654/2020-
AR-VR-Survey-v3.pdf.