Trustworthy Swarms
James Wilson Greg Chance
University of Bristol and Dyson
Institute Of Engineering &
Technology
UK
Suet Lee
University of Bristol
UK
Shane Windsor
University of Bristol
Bristol, UK
Jonathan Ives
University of Bristol
UK
ABSTRACT
Swarm robotics has begun to move from the laboratory to the real
world. Potential applications include: logistics, environmental mon-
itoring, search and rescue, and medicine. However, a key challenge
for the deployment and use of these systems is the interrelated
factor of ensuring technical, human and societal trust. To help ad-
dress this complex issue, we first turn our attention to research on
the technical properties of swarms, such as proficiency, scalability,
robustness, and adaptability and their role in building trust. Second,
we explore the area of research known as Human-Swarm Interac-
tion, which studies how humans understand, monitor, control, and
interact with swarms. Third, we focus on techniques used to specify,
verify and validate swarms. Finally, we discuss the future of swarm
robotics and conclude by suggesting areas of advancement which
may help build human and society’s trust towards robot swarms.
KEYWORDS
Swarm Robotics, Swarm Engineering, Trustworthy Autonomous
Systems, Human-Swarm Interaction, Swarm Interfaces, Automatic
Design, Verification and Validation, Safety, Fault Detection
ACM Reference Format:
James Wilson, Greg Chance, Peter Winter, Suet Lee, Emma Milner, Dhaminda
Abeywickrama, Shane Windsor, John Downer, Kerstin Eder, Jonathan Ives,
and Sabine Hauert. 2023. Trustworthy Swarms. In First International Sym-
posium on Trustworthy Autonomous Systems (TAS ’23), July 11–12, 2023,
Edinburgh, United Kingdom. ACM, New York, NY, USA, 11 pages. https:
//doi.org/10.1145/3597512.3599705
This work is licensed under a Creative Commons Attribution International
4.0 License.
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
© 2023 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0734-6/23/07.
https://doi.org/10.1145/3597512.3599705
Peter Winter
University of Bristol University of Bristol
UK UK
Emma Milner Dhaminda Abeywickrama
University of Bristol University of Bristol
UK UK
John Downer Kerstin Eder
University of Bristol University of Bristol
UK UK
Sabine Hauert
University of Bristol
UK
1 INTRODUCTION
Swarm robotics is the study of multi-robot systems that exhibit
global emergent behaviours through local environmental and robot-
robot interactions. Over the past two decades, there has been a
significant surge in swarm robotics research, and its outcomes
are just beginning to translate to real world applications [25, 67].
Making a successful jump from the laboratory to the real world
requires the development of trustworthy swarms: swarms that
human developers and wider society can trust [18].
Although research into Human-Swarm Interaction has started to
collect views on the development, deployment and use of swarms,
most of this work has focused on the views of users and expert
participants (so called user studies) and is often limited to laboratory
experiments. Human-Swarm Interaction research also tends to be
limited to quantitative methods (largely surveys or questionnaires)
which overlooks rich qualitative descriptions of their interactions
in the development of these systems as well as wider human work
practices riddled with political and organisational contingencies [15,
18, 24, 47, 79]. This narrow focus has also meant that expert views
have dominated the views of the public at large, with members of
society continuing to remain a “missing voice” in the development
and implementation process. Taken together, pertinent questions
remain on the design, development, deployment and usefulness of
robot swarms. As robot swarms become much more of a reality, such
questions have received growing attention by developers, scholars,
policy makers, regulators and the public at large to understand
the co-constitution of robot swarms and trust. Such questions are
important to address if we are to consider the journey towards
acceptance or use of this technology.
This article does not attempt to flesh out human views or at-
titudes on swarm robotics. However, it does lay the ground for
technological and empirical work that can help in designing trust-
worthy robot swarms. To begin this process of mapping out areas
for research on human trust in swarm robotics, we first look to
science fiction narratives to get a sense of future worlds which did
not pay enough attention to the design of trustworthy swarms.
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom Sabine Hauert et al.

It has long been said that depictions of emerging technologies

in science fiction influence society in some form (our emotions,
ambitions, values, neuroses, conflicts, passions, and politics) and
play a role in shaping our imagination. Such narratives can act
as powerful rhetoric and site of personal politics which may or
may not lead to the rejection and distrust of such systems [59]. For
example, the film Big Hero 6 (2014) [2] illustrates how a swarm
of microbots performing complex construction and transportation
tasks may lead to or result in the misappropriation of such robots
by “bad” actors. The short film Slaughterbots (2017) [77] portrays
a similar message through the use of a multi-robot swarm system
that is capable of targeting and killing humans at huge scale with
relative ease. Other dystopian swarm futures can be seen in series
like Black Mirror, or science fiction books such as Prey or the Silo
Trilogy. Taken together, robot swarms are imminently portrayed
as “problem” technologies that affect people and society in harmful
ways.
Harmful robot systems like these can be intentionally or uninten-
tionally developed by human actors and/or organisations because
they are shaped by a wide variety of human, social, cultural, po-
litical, and organisational factors [57]. Taking these factors into
account, human developers must undertake due diligence when
developing swarms. Given the possibility of intentional or unin- Figure 1: Ongoing research in areas of swarm robotics with
tentional harm, swarm developers and their organisations face potential impact on the development of trustworthy swarms.
significant challenges in building robot swarms that are trustwor-
thy, safe, and accessible to all communities in applications such as
seeking to specify, verify and validate swarm systems with emer-
manufacturing, logistics, and search and rescue.
gent behaviour.
While there is a gap of knowledge on human views about robot
swarms, developing and using robot swarms brings along further 2 SWARM PROPERTIES
empirical, theoretical and practical questions about the dynamics
of robot swarm adaptability. We may imagine public concern sur- A major part of the daily work of swarm engineers is in the de-
rounding the swarm’s emergent behaviour and how it might change velopment of technical properties that are associated with swarm
from an expected one to an unexpected one due to its interactions systems. In line with the literature on swarm robotics and trust, we
with the environment. Put simply: a robot swarm’s capacity for identified four main properties of swarm systems that developers
adaptive and unpredictable behaviour may undermine human trust. are particularly interested in when it comes to swarm development.
These concerns point towards potential problems of human un- These properties are: proficiency, robustness, scalability, and adapt-
derstanding when it comes to human-swarm interaction. Problems ability. Such properties of swarm systems and their effect on human
of understanding arise when humans interact with a complex sys- trust is outlined below.
tem whose global behaviour emerges from the interactions amongst
multiple individual robots. As a human, observing these interactions
2.1 Swarm Proficiency
and attempting to make sense of system goals can be incredibly “Proficiency” relates to the swarm’s ability to fulfill its task with high
difficult, consequently such systems may be perceived as unpre- performance. System performance has a large impact on human
dictable [18]. Predicting how individual robot behaviours might trust [33], in that a well-working system may be seen as more
impact global behaviour has been a challenge within swarm robot- trustworthy than one with poor performance. This is especially
ics for some time [32] and has given rise to research that aims to the case if performance can be maximised without impacting other
make swarm behaviour more understandable to their developers features that contribute to trust. However, this is usually not the
and users [11, 44]. case. Increasing the maximum available speed of a robot performing
As outlined in Figure 1, this paper highlights ongoing research a logistics task might increase its delivery performance, but may
in areas of swarm robotics that may have an impact on the devel- decrease the safety of the system and in turn contribute to distrust of
opment of trustworthy swarms. We first focus on existing research the swarm. Conversely, a stationary robot performing no function
about design properties of swarms, such as proficiency, robustness, is certainly safe but not very helpful.
scalability, and adaptability, which may support or hinder human
trust. Second, we visit the literature on Human-Swarm Interaction 2.2 Swarm Robustness
and pay attention to how social interactions relate to the build- The property of “robustness” often relates to the expendability
ing of trust towards swarms. The third section on verification and of individual robots [14]. In swarms, individual robots will often
validation delves deeper into the question of how developers are fail with no substantial impact to the task the overall swarm is
performing. As a result, focusing on the robustness of the swarm
Trustworthy Swarms
may help to demonstrate high proficiency in a task, and thus build
trust towards the overall swarm. However, while a swarm that is
tolerant to multiple failures might perform a task proficiently, a
human user may still observe faults taking place. Subsequently, the
user may perceive that the capabilities of a faulty individual are
representative of the whole swarm, impacting the user’s perception
of the swarm’s trustworthiness.
The failure of individual robots within a swarm can sometimes
do more damage than merely altering a human user’s perception
of the overall system. While it is assumed that a failed robot will
have little negative impact on the swarm, in some instances, failed
robots may have significant impact on the swarm performance. For
example, a single robot may restrict or “tether” the entire swarm,
causing other robots to fail at their task [12]. Research into these
types of failures must be sought in swarm engineering in order to
be mitigated or avoided. Deploying systems with weaknesses in
robustness can shape a human’s negative perception of the swarm
and diminish public trust in swarm systems.
2.3 Swarm Scalability
The replication and scaling up of individual robots in a swarm
could, in theory, improve its overall performance and encourage
humans to trust the system. However, increasing the number of
robots within a system comes with increased complexity for a user
as it becomes more difficult to cognitively process what each robot
is doing in parallel. In addition to this, increased performance with
increased robot numbers is not always guaranteed. In most systems
there will be a maximum performance attainable by increasing
swarm size, meaning that there will be a point at which performance
will no longer continue to increase and will most likely decline as
robots are added to the system [38]. Such a drop in performance
could result in distrust, especially by people who are in the process
of developing the system and those who are operating or interacting
with the system. Increasing the scalability of the swarm may also
make it difficult for humans to assign clear accountability to specific
robots which fail or cause harm among the swarm.
2.4 Swarm Adaptability
Swarms and the algorithms that dictate their behaviour can result
in “adaptability”. The use of algorithms that dictate pheromone
navigation [17], for example, have been shown to generate adap-
tive properties in swarm systems. Such algorithms are said to or-
chestrate swarm navigation in terms of finding the shortest path
between two points through continuous travel. Paths are adjusted
by the algorithm in the event of obstacles that block the path of the
swarm. In some cases, the swarm may even identify multiple paths,
splitting the traffic across two routes. This is especially the case
if the swarm is too large to travel along the single shortest route
without congestion [66].
In recent research, we see discussions on how robot diversity
amongst heterogeneous swarms can lead to systems being signif-
icantly more “adaptive” [39]. A good example of this can be seen
in [80]. Wilson’s methodology shows a heterogeneous swarm ex-
ploring a diverse environment. The swarm leverages its differing
morphology and adjusts the preference of individual robots through
the use of a hormone inspired system. The resulting effect means
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
that the individual robots self-organise themselves to environments
that are best suited to their material and algorithmic properties.
The adaptation in the systems we discuss here stems from the
quantity of robots within the swarm. Robots can test alternative
methods or compare their performance with one another directly
or indirectly, gradually adapting to changes in environment or task.
Such adaptability to volatile or unpredictable environments may be
seen as a positive element when considering trust, or may make the
swarm seem unpredictable, resulting in lower trust of the system.
3 HUMAN-SWARM INTERACTIONS
Human trust in swarm systems has not yet been studied in great
detail. Robotics research often lacks qualitative case studies on
the configuration and influence of technical tools or technologies
for technology-mediated knowing. When qualitative research is
sometimes produced as part of their research design, it tends to be
subsumed under technical and engineering detail. For these reasons,
it is difficult to conclusively say what factors help to build trust, or
whether actively faulty robots will cause issues of mistrust. In the
next section, we draw attention to the literature on Human-Swarm
Interaction in order to give us a better picture of how humans
and swarms interact, with a focus on how humans can monitor,
understand, and control swarms.
3.1 Understanding Swarms
3.1.1 The Challenge. A substantial source of distrust in swarm
systems may stem from the complexity and unpredictability of a
swarm’s behaviour. Unpredictability can generate gaps and uncer-
tainties in human knowledge which makes it increasingly difficult
for us to understand the motive or goal of a swarm. In addition, the
distributed nature of a swarm challenges developers in terms of
knowing what information is actually valuable and how to present
this information to humans, placing greater emphasis on estab-
lishing meaningful linkages between humans and robot swarms.
Unpredictability, in particular, has already been noted as a large
factor in human distrust of autonomous systems [69] and the wider
field of Human-Robot Interaction (HRI) more generally [10].
These problems highlight the need for some form of understand-
ing between human operators and swarms. The Human-Swarm
Interaction literature typically highlights the importance of human
operators as being able to make observations and predictions about
the swarm, allowing them to understand system behaviour: what
the system is doing, why, and what it will do next [61]. The chal-
lenge here, then, is to develop tools that allow human operators
to easily understand the state of the swarm, its performance, and
its emergent behaviours across many different tasks or problems
[15]. Such tools may also be used to inform humans of potential
collisions, and faulty or defective robots, as well as to allow enough
time to stop accidents from happening [20]. If a swarm has been
developed without some form of human understanding of what the
system is doing, it becomes difficult for human operators to trust
the system and predict its behaviour.
3.1.2 Operator interactions: monitoring swarms. Building a trans-
parent system in terms of its observability and predictability is a
good starting point to understand what the swarm is doing, why
and what it will do next. For example, it can be argued that the
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom Sabine Hauert et al.

swarm will produce a substantial amount of information about its adapted and re-ordered, and additional sub-behaviours can be in-
behaviour, allowing human operators some insight into what is cluded [23]. In this instance, behaviour trees represent a human-
going on. Yet, given the swarm’s distributed nature and adaptability readable form for helping to understand robot behaviours more
which, in effect, lead to a substantial increase in different types of effectively than they would be able to with traditional black-box
interactions and unpredictable behaviour, human operators will style neural network controllers [43]. With a more human-readable
find it increasingly difficult to be aware of everything and process format, humans interacting with a swarm can gain insight into
this information. In this instance, too much information might in individual robot behaviour, which may make the swarm more inter-
fact be harmful, confuse and overwhelm operators, or even erode pretable and provide understanding of such behaviour. Equipping
their trust, especially in high risk settings. Reiterating the impor- operators with such visual information may help to build a degree
tance of system transparency, information about the swarm needs of trust towards the system and, during operation, could enable
to be human “readable” which can help make swarm functionality them to make better decisions about whether to continue to trust
less opaque, and actionable. “Actionable” in the sense that the oper- the system. Behaviour trees and the issue of control will be covered
ator has knowledge about the swarm performance, can change the in more detail later in this article in the section: “Swarm Control”.
behaviour of the swarm to improve performance, and can detect
3.1.3 Public Interactions: Adapting to swarms. In the previous sec-
and mitigate faults.
tion we highlighted a variety of methods for monitoring and un-
This matter of monitoring and controlling swarms becomes even
derstanding the emergent behaviour of swarms. Specifically, we
more challenging outside controlled laboratory settings in real
outlined a variety of interfaces swarm developers are designing to
world swarm scenarios. Take for instance a swarm having to detect
establish effective, meaningful and interpretable interactions with
or extinguish a forest fire, the number of robots involved, dynamics
their specific swarm system (e.g. [24, 40, 69]). However, while there
of the fire and environment, and high-pressure operation are likely
are great efforts underway to improve human-swarm interactions,
to have adverse impacts on the “cognitive workload” of human
most of this research is geared towards human users and operators
operators [40, 47]. For example, according to Kolling et al., [47],
(operators who may even have a role in developing the swarms).
the monitoring of a large number of robots can create an over-
This means there is a lack of research on human-swarm interactions
whelming level of “cognitive complexity” for an operator due to
from a public perspective. Nor is there much research that tackles
the observation of robots moving in different directions.
the challenges of public(s) and different cultures adapting to real
Consequently, there has been a surge in human-swarm interac-
world applications of swarms [37]. Initial work exploring human-
tion research on the management of human cognitive workload
swarm interaction in public settings has focused on art, outreach,
and control strategies [56, 62, 68]. For example, Nunnally et al. [62]
and opinion polling [8, 9]. The identification of meaning-making
realise how the display of certain interface-related elements – such
processes from a public perspective may contribute useful knowl-
as a viewport of the open environment and swarm state displayed
edge to human-swarm interaction research, which could inform
to the human “from a birds eye, orthographic view” – function
more effective trust practices in understanding the behaviour of
as props for operators monitoring their swarms. Other attempts
swarms. For instance, research on the adoption of Autonomous
to simplify swarm information for human operators include: heat
Vehicles (AVs) has identified the value of external human-machine
maps displaying the belief and confidence of received swarm in-
interfaces (eHMIs) as a solution to the loss of social interactions
formation [24], augmented reality systems providing live robot
by human drivers (such as hand gestures or walking behaviour),
information and gesture-based swarm control through haptic in-
opening possibilities for research on meaningful interface design
terfaces [7, 20], as well as interfaces that display the percentage of
that could help make these vehicles safer for the public (as well as
swarm distribution defined in terms of the operator’s phenomena
safe for the driver) (e.g., [31, 49, 58]). Here, members of the public
of interest [36].
are invited into an inclusive design process that aims to put their
Similar research has provided human operators the ability to
voices and lived experience at the heart of the design process. For
learn swarm dynamics through the use of automatically gener-
example, Swedish company Semcon 1 have put forward the idea
ated explanations which make explicit the decisions made by au-
of a ‘smiling car’ so that the human is likely to understand the
tonomous robots [76]. This work found that explanations of swarm
vehicle’s intent: when the vehicle detects a pedestrian a “smile”
behaviour helped build transparency and trust in autonomous
lights up on the front grille confirming that it will stop. Elsewhere,
robots. By giving operators the option to inspect the decisions
Ackermans et al. [5] are developing a type of LED light strip to
of individual robots, robots that were deemed to have made poor
support the interaction between AVs and pedestrians through the
decisions in the past were no longer ignored by operators due to
use of colours or animations to express the vehicles intention, with
their prior failures. Giving operators an explanation of the deci-
members of the public having a preference for “uniformly flashing
sions made by the robot helped to build trust in poorly performing
or pulsing animation” when AVs communicate a yielding intention.
systems.
Taking the AV research into consideration, if swarms are to be
Elsewhere, other research has addressed problems of human-
located in the public arena, then new modes of interface design
swarm interaction through the adoption of behaviour trees - a hier-
will be required for public interaction. If this is the case, a combina-
archical method for visualising the system’s emergent behaviours
tion of these insights can help inform a design response to better
[43]. Behaviour trees are formed from modular branches displaying
facilitate public interactions with swarms. This may generate a
behaviours and sub-behaviours. These behaviours can be easily
variety of new interaction questions: How much information does
1 The Smiling Car - Self driving car that sees you - https://semcon.com/smilingcar/
Trustworthy Swarms
a member of the public need to know, and to convey information
with clarity that is immediately recognisable to the swarm system?
How can multiple swarm systems from different companies (who
have different technical, political and commercial interests) operate
harmoniously and interact with each other (and their public) in
society?
3.1.4 What’s Next for Human-Swarm Interaction? The previous
sections have highlighted how human-swarm interactions influ-
ence trust in the behaviour of a swarm system, and what could
cause distrust and uncertainty. Future Human-Swarm Interaction
research should develop methodologies for collecting swarm infor-
mation and visualising swarm properties, and deliver this informa-
tion in a human readable format that is interpretable to the user.
However, the balance between the visualisation of information
and improving operator performance without increasing opera-
tor workload remains a challenge [7]. In addition to interfaces, a
human-understandable answer of what the swarm is doing can be
mediated by the design of the swarm system itself. As previously
mentioned, there are methods through which swarm behaviour
and configuration can be designed or evolved, which make systems
more human readable. This, in turn, will help make the job of in-
terfaces conveying swarm intent simpler and easier to understand.
Although as work by Walsh [75] reminds us, this is easier said than
done as designing human-swarm interactions requires a good un-
derstanding of how people think, their preferences, and the nature
of their day-to-day work.
Furthermore, it is also important to consider the type of train-
ing human-operators and users require in using swarm systems.
Appropriate training not only helps in the responsible use and man-
agement of swarm behaviours (e.g. fault detection), but it may also
prevent cognitive overload and manage stress as well as improve
the adoption of the swarm in workplaces or wider society. More-
over, it will be important to consider who will be consuming swarm
information. For example, in the context of professional work, a
warehouse operator is not going to require the same information
as a nuclear decommissioning engineer. Developers who recognise
such requirements will encourage adoption and potential trust in
swarm systems. However, the use of swarms in contexts of pro-
fessional working environments can also be associated with other
undesirable ramifications including complacency, deskilling of pro-
fessional expertise and the degradation of work in the long term
[40, 41, 51]. For example, periods of monitoring swarm systems may
undermine skills and diminish operators’ ability to generate expec-
tations of correct behaviour. Matters such as deskilling (as well as
“overtrust” and the tendency for humans to over rely on swarm
information) require further attention and must be addressed if
operators and society are to fully trust and accept swarm systems
in the future [51].
3.2 Swarm Control
3.2.1 The Challenge. Some swarm systems, particularly those in-
teracting closely with humans, require an element of human control.
Human input can benefit the swarm’s output by providing context
for the activity that is not available to the swarm. However, when
it comes to providing context and understanding of the activities
of the swarm, the volume of robots within a swarm can make this
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
challenging. As we have already discussed in earlier sections: multi-
ple individual robots can complicate the human operator’s decision
making processes (their cognition abilities are harmed when mon-
itoring, for example [55]). Not only this, but in centralising the
swarm to a human operator and visualising a single point of failure,
the robot system may lose its swarm-like properties. This concept
has created a large field of research, giving insight to a multitude
of control options that allow humans to interact with swarms.
Human-Swarm interactions can take the form of control at vari-
ous resolutions, from manual control (i.e. control over all individ-
ual robots) to decentralised control (i.e. control that preserves the
swarm’s distributed and emergent nature). Prior work often refers
to these forms of control in terms of “Levels of Autonomy” (LOA).
According to the literature, LOA tends to fall into three categories
[61]:
Manual: Total human control of the swarm system.
Mixed-initiative: A hybrid approach in which swarms and
humans collaborate.
Fully autonomous: A swarm which acts with no human in-
put to behaviour.
Existing research presents arguments for both hands on and
hands off approaches. An argument for a fully autonomous swarm
can be seen in work by Walker et al. [74], which coins the term “ne-
glect benevolence” in reference to the idea that humans can benefit
from reducing their input, and leaving full autonomy to the swarm.
However, Human-Swarm Interaction studies have also found that
human intervention can be key to success in some scenarios. Such
interventions are said to alleviate shortcomings in autonomy and
concerns around changes in the intent of the swarm [47]. Finding
a balance between these options will most likely change because
of differences in work practices and other social, cultural, and or-
ganisational factors. We note here that swarm developers will need
to work closely with operators/users of these swarms in order to
establish preferred levels of autonomy and interactions when these
systems are deployed in work environments.
3.2.2 Tools for swarm control. In this section we focus attention on
different tools that are being developed that allow human operators
to control robot swarms.
First, research into operator preferences for the control of robot
swarms reveals a strong preference for Swarm-User Interfaces. For
example, a study by Kim et al. [45] reveals a preference for “tangi-
ble” user interfaces which provide tangible links between human
operators and the real world, allowing specific observations and
interactions to be linked with a variety of control commands such
as gesture, voice, or physical touch. Such modalities are seen as a
means to establish effective control of swarms through user-defined
interaction. In this work, gesture was found to be the most com-
monly used interaction (66% of the time) followed by verbal inter-
action (26% of the time) and then touch (23% of the time) for swarm
control. Despite the benefits that these interaction-methodologies
bring, the authors also noted a number of shortfalls. For example,
operators had a strong preference for gesture and touch based in-
terfaces, yet the usefulness of these interfaces are likely to diminish
in dark rooms because such vision systems were designed for light
or bright environments. In addition to this, a sparse swarm spread
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
out across a large environment would not be in view of the opera-
tor. Although gesture or touch may not be useful for total control
of a system, it could function well as a method of control for a
semi-autonomous swarm.
A similar approach on swarm control considers the design of an
augmented reality human-swarm interface [64]. In this example a
human operator was equipped with a tablet-based augmented real-
ity interface allowing the human operator to virtually select, handle,
and drag objects to their intended destination within a digital twin
of the physical environment. In this instance, the augmented reality
application allowed human operators the opportunity to control
large numbers of robots by focusing their gaze on the movement of
virtual objects (“environment-orientated action”), rather than the
many robots of the swarm (“robot-orientated interaction”). Interest-
ingly, Patel et al. [64] found that virtual object manipulation through
environment-orientated commands significantly reduced the cog-
nitive load of human operators. This suggests that human control
of swarms may be effective if the operator is focused more towards
environment-orientated interactions and swarm-wide goals rather
than robot-orientated commands.
Second, there is a growing interest in designing behaviour trees
as a strategy for controlling robot swarms. For example, Hogg et al.
[35] present a methodology for generating human-readable control
for swarm behaviour through the use of behaviour trees. In this
instance, behaviour trees are artificially evolved to generate human
understandable and human readable swarm behaviour options. In
theory, this methodology should allow human operators to bet-
ter interpret the behavioural options available to the swarm. In
turn, this allows the operators to pass high level commands to the
agents of the swarm, providing influence while preserving swarm
properties.
Importantly, the technique of artificial evolution has been a pow-
erful tool within the context of swarm robotics because of its ability
to discover robot controllers resulting in desired swarm behaviours
[13, 48]. Like the example above, we can see how evolutionary
algorithms can generate bespoke user specification and discover so-
lutions the engineer might not have thought of, such as commands
or controls that a human can control or design. By iterating through
many generated designs, evolutionary algorithms can identify not
just one strong solution to a problem but a multitude of solutions.
Multiple-solution evolution has been used in novelty searches [53]
and more recently in the MAP-elites algorithm [27, 60] which gen-
erates a large number of specialised high performing solutions for a
wide range of characteristic sub sets. These methods provide human
operators who interact with swarms with pre-evolved high per-
forming options, among which they can choose to fit their specific
needs. Human operators can also effectively edit swarm behaviours
ad hoc as attitudes or tasks change in a given scenario. For example,
in a warehouse logistics scenario - where the job of the swarm is
to collect items, store them, and deliver them back upon request -
human operators could choose a swarm configuration that priori-
tises energy efficiency over the number of items delivered over a
certain period of time. In this way, users are given the opportunity
to calibrate and optimise swarms for the characteristics they per-
sonally value, including the configuration they trust the most, in
their specific application of the system [79].
Sabine Hauert et al.
The above three approaches are a step towards prioritising hu-
man operators’ preferences, a principle that is recognised as a core
requirement of a desirable and trustworthy technology [6]. How-
ever, this also raises the concern that efforts to integrate users’
preferences in the design of swarm systems may end up shifting
responsibility from swarm developers onto the human operators
who have expressed those preferences. In this way, the operator
could become a “moral crumple zone” [26]. Like crumple zones in
cars are designed to crumple in case of a crush and so absorb some
of the energy of the impact, this term is used to refer to human
operators or end users who end up bearing the moral and legal
responsibility for the malfunction of a complex autonomous system,
despite having little control over such system.
This concept applies to cases where swarm behaviour has been
engineered to provide operators with multiple performance options
[79]. Even if operators are given the option to choose their preferred
swarm configuration, their choice is limited to the finite number
of configuration options pre-designed by the developers. On the
one hand, this means that these configurations can be individually
verified prior to use, creating greater reassurance that the system
will perform as requested. On the other hand, the implication is that
developers do make critical choices in terms of swarm performance
and safety at the stage of system development and verification, and
so could hold some level of responsibility in the event of system
failure. These considerations highlight the need for the development
of clear rules on accountability and liability if these “personalised”
robot swarms are to be adopted, and trusted, in the real world.
These examples highlight preferences for Human-Swarm Interac-
tions as well as methods for making these interactions more visual
and meaningful to human operators. However, a key challenge
remains: how to identify the right amount of human interactions
that are optimal with the swarm (in the sense that they maximise
trust).
3.2.3 What’s Next for Swarm Control? Having explored some ap-
proaches to the control of swarms and highlighted how interactions
may affect trust in swarm systems, it seems that Human-Swarm
Interaction is something that will be required should swarms be
trusted to operate in mainstream settings. At the bare minimum, a
human should have the ability to deactivate a system that operates
unsafely. In addition, swarm systems that are being deployed in
the public domain should also have some means of engaging in
some form of meaningful interaction. For example, how should a
swarm react to the presence of a human? How should a swarm
behave in a safe manner and ensure that no harm comes to those
in its proximity? Having identified human-swarm safety as a key
component in building trust, the next section builds on this concern
with safety by focusing more specifically on technical methods that
help developers/operators perform quality control and manage the
safety of swarm applications.
4 SWARM TESTING
Before robot swarms are deployed in the real world they need to be
tested in a way that follows accepted technical standards in swarm
engineering. Such testing methods are designed to help developers
build safety cases for swarm robots in order to make these systems
pass safety concerns before they are deployed in society. Examples
Trustworthy Swarms
of swarm applications in which swarm safety is crucial include
search and rescue [65], construction [78], and space exploration
[67, 73].
Hunt and Hauert [37], in particular, have provided a “safety
checklist” for swarm engineers to consider a number of safety-
related items that range from ethics and regulations, to the possibil-
ity of physical or behavioural harm from a swarm. As a swarm is a
decentralised and complex system, it is hard to know whether the
emergent behaviour from individual interactions will be safe. This
is one of the main challenges we face in guaranteeing swarm safety.
In the following subsections, we focus on two popular methods that
swarm engineers are using to develop safe swarms: fault detection,
diagnosis and recovery, and security.
4.1 Fault Detection, Diagnosis and Recovery
Existing Fault Detection, Diagnosis and Recovery (FDDR) methods
typically address systems simpler than swarms [63]. Methods of
fault detection and diagnosis may be classed as either endogenous
or exogenous: the former means that a robot detects and diagnoses
its own state, and the latter refers to a robot detecting the state of
others. A single robot system is only able to perform endogenous
fault detection by definition, but a robot in a swarm may perform
both. Graham Miller and Gandhi [30] present a survey of exogenous
fault detection and diagnosis methods. In the survey, fault detection
methods are categorised as data modelling-based, immune system-
inspired or blockchain-based, whereas fault diagnosis methods are
categorised as local sensing-based. As a point of interest, Graham
Miller and Gandhi [30] point out the work of Christensen et al. [22]
on firefly-based exogenous fault detection and claim this to be the
beginning of swarm specific work on fault detection. They go on to
highlight how the immune system is analogous to a swarm system
and consider exogenous fault detection as an artificial immune sys-
tem model for fault detection in engineered systems [63]. Similarly,
Tarapore et al. [71] present a fault detection method based on the
adaptive immune system of vertebrates.
Additionally, an immune-inspired fault recovery method is pro-
posed by Ismail et al. [42] where robots may “self-heal” through
simulated trophallaxis (a term borrowed from behavioural biology
to describe the mutual exchange of food between adult social insects
or between them and their larvae) to take power from failed robots.
Beyond immune inspiration, blockchain-based methods may be
used to detect malicious or faulty robots in a swarm. Strobel et al.
[70] propose a blockchain protocol where a single robot must per-
form “proof of work” in order to prove that its sensors adequately
work before sensor information is shared to the collective. Although
blockchain methods do not specifically detect faults, blockchain
approaches can be used to diminish and block the contributions of
faulty or malicious robots [30]. Finally, data-driven methods have
also be explored to systematically extract local metrics to detect
faults in robot swarms [52].
It would be useful to consider the FDDR framework holistically as
each step should inform the others in an iterative process. Bjerknes
and Winfield [12] demonstrate that the assumption that a swarm
is robust is false. If this is the case, engineers should decide on the
number and severity of faults that should be tolerated in the system.
On the other hand, a swarm may prove robust to malicious attacks
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
in the context of cybersecurity due to its decentralised nature. These
examples of robustness illustrate how unique the problem of FDDR
is in the swarm context.
5 VERIFICATION FOR SWARMS
Verification and Validation (V&V) is a technical process that engi-
neers use to gain confidence in the correctness of a system relative
to its requirements. Prior to and separate from verification, a tech-
nical specification of the system must clearly define the expected
operational behaviour of the system and many challenges are asso-
ciated with this task for autonomous systems [3].
Recent work by Abeywickrama et al. [4], in particular, proposes
a new process for the safety assurance of emergent behaviour in
autonomous robotic swarms called AERoS (Assurance of Emergent
Behaviour in Autonomous Robotic Swarms). Focus is given to spec-
ification design as a means to assure the safety of the emergent
properties of the swarm, rather than the individual behaviours of
the robots [4]. Given the specifications, there are different verifica-
tion methods that can be employed to test the safety of the system.
Such methods are often chosen by those involved in the design
and development process in order to better fit the relevance of the
application, operational domain and the specific properties that
need to be verified.
5.1 Swarm Trustworthiness Properties
To continue assessing the trustworthiness of swarms, distinctive
properties of the swarm, such as proficiency, adaptability, robust-
ness, and scalability may require methods that extend beyond the
traditional model of V&V (i.e., requirements, design, implementa-
tion, testing and deployment). Since trustworthiness is a property
that encapsulates not just technical safety and functional correct-
ness but many other non-functional attributes, for example, social
interactions and ethics [28, 50, 82], those involved in the design
and development of swarm systems must explore more holistic
approaches in order to consider the social and ethical implications
of the technology.
A major challenge in the verification of this holistic and so-
ciotechnical approach is the lack of standards and regulations by
which they can be evaluated against. While technical verification
methods exist to help developers assess what the technology should
do (i.e. in terms of its functionality), a pertinent question remains as
to whether such strategies - typically focused on the technical prop-
erties of the system - are suitable for the social, ethical, regulatory
and legal dimensions of testing. Technical methods of verification
and validation can only go so far in proving whether a system
is “safe enough” before it is released into the wild. For example,
while there are well defined rules to determine driver conduct [72]
which is often verified using technical methods of simulation-based
verification [34], the standards for evaluating not-so-static social
interactions or ethical behaviour are either impossible to capture
or extremely difficult to model due to the dynamic nature of ev-
eryday activity and the contextual relevance of the ever changing
social environment [29]. Such consideration of human and social
factors are important if we are to build and maintain user trust in
the system [21], especially in the context of different moral and
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
epistemic articulations of “under trust” or “over trust” which can
lead to serious and sometimes fatal consequences [46].
5.2 Swarm Trustworthiness Properties &
Assessment
Robustness, scalability and adaptability are considered as being par-
ticularly useful properties of swarm trustworthiness, although this
list could be extended to cover other properties 2 . These properties
are examined in regards to swarm trustworthiness.
5.2.1 Trust Property: Robustness. “Robustness” is often described
as the degree to which a swarm system can function correctly in
the presence of failed robots. Failures may come in many forms,
from hardware and software failure, to malicious actors subverting
the behaviour of a robot. A suitable verification task may be to find
the limits of robustness either through simulation software and/or
hardware trials of whether a task is successful (or not) in order
ensure that the task in question meets the specifications of both
human developers, operators and/or users. For example, does the
swarm still successfully complete assigned tasks when a specific
number of robots fail or are removed from the swarm?
5.2.2 Trust Property: Scalability. “Scalability” is defined here as
the ability of the system to complete assigned tasks with a larger
number of robots, i.e. “scaling up”. This property of scalability is
somewhat integrated into the swarm engineers’ conceptualisation
of a swarm and has become a key parameter in the optimisation
of performance and other non-functional properties dependent
on the specific application or context. Scalability is also central to
the development of metrics that provide a quantitative measure
of coverage and spatial self-organisation. For example, established
metrics such as “spatial-density” (i.e. the amount of space that is
provided per robot within its own area), could be used to develop
a measure of trustworthy density. We imagine the metric of “trust-
worthy density” to be useful to human operators if they need to
assess whether their swarm system requires a minimum or max-
imum density of robots to perform optimally. Such a metric may
also be used by the swarm to self-select its own volume based on
“operational feedback”, where the number of idle robots may be
considered surplus to needs, or conversely over-utilised and require
more robots to efficiently complete assigned tasks. Another scalabil-
ity metric could be designed to focus on performance and may also
include other metrics, such as task completion rate and network
connectivity. In theory, metrics such as these could help engineers
build trustworthy swarms. This interplay between density and per-
formance can be crucial, especially when the completion of a task
favours a low density of swarms (allowing robots free movement)
but network connectivity is optimal at the highest density (highest
network redundancy).
As we have previously mentioned, coordinating a large group
of robots may cause human operators to become overwhelmed
with information, including metrics or data that may unhelpfully
distract the operator and potentially lead to unsafe situations. In
this instance, we repeat our previous argument early in this article:
2 Trustworthy Systems Lab - Trustworthy Autonomous Systems
https://github.com/TSL-UOB/TAS-Verif
Sabine Hauert et al.
Human-Swarm Interfaces should be designed to ensure that human-
swarm interaction is concise and readable irrespective of swarm
scale. In addition, user experiences should not only be collected,
but also readily accessible and shared within organisations and
across entire industries as a means to improve safety [57]. The
operational experiences of operators or users interacting with these
swarms may be collected in the form of interviews, questionnaires
or surveys as a means to effectively explore the safety implications
of human-swarm experiences, preferably during developmental,
testing, or operational activities [57].
5.2.3 Trust Property: Adaptability. “Adaptability” is the degree to
which a system can effectively and efficiently adapt to different
or evolving environments 3 . For a swarm application, this could
be operating with reduced robot numbers or environmental ob-
structions, such as a blocked path in a warehouse creating the need
for another route. In this case, effective metrics could be based
around robot “idle times” (time not spent doing something useful)
and “recovery rates” (speed at which a new route is found) when
the swarm encounters obstructions. Similarly, performance metrics
could be used to assess the adaptability of swarms performing in
some particular environment - such as “task success rate” in a find-
and-fetch operation in relation to some particular problem, such as
determining its adaptability in a new building layout.
5.3 Swarm Assessment Methods
Formal verification can be used to prove whether a swarm property
is consistent against an entire parameter space. This is in contrast
to sampling the parameter space if using a testing method, such
as a simulation technique. While formal verification can help pro-
vide proof of assurance against single properties of a system, the
technique is not always suitable for complex systems operating
in unbound domains, with a high dimensional parameter space.
However, a swarm system operating in a bound environment with
a fixed number of actions may be more easily verifiable in practice
compared to a self-driving vehicle. In this case, a swarm of robots
that use a fixed set of actions - such as behaviour trees - is perhaps
better suited to formal verification. For example, formal methods
such as Probabilistic Finite State Machine (PFSM) verification have
been used to demonstrate the robustness of a swarm even when
network connectivity is lost in terms of local network failure [81].
Formal verification methods can help build notions of resilience
and assurance against a large range of parameters. However, such
methods only give assurance of the model and there may be distinct
difference between the model and the real system. This is where
testing methods, such as simulation based verification, can help
build evidence by bridging the gap between simulation and reality
in making assurance arguments.
A corroborative approach to verification and validation, in partic-
ular, seeks to combine mutually consistent evidence from multiple
and diverse assessment methods that will raise the confidence of en-
gineers in terms of their system’s trustworthiness [34]. For example,
a combination of both physical and simulation-based verification
methods can be used to help build a test suite. [34], claim that their
- 3 International
Organization for Standardization - ISO/IEC/IEEE 24765:2017 Systems
and software engineering — Vocabulary - https://www.iso.org/standard/71952.html
Trustworthy Swarms
approach is useful because it complements both the reliability of ex-
isting formal assessment methods and the bounds of the operational
domain, such as edge cases.
Many verification and validation techniques can be used to build
a suitably diverse test suite. These techniques may include manual
written test cases by humans which are based on historical evi-
dence (“accidentology”) [1], or fault-based test selection, based on
requirements coverage and robot-based test generation [19]. There
are also assertion-based techniques that can be used to prove that
the system complies with a set of provided rules, such as the driving
conduct of self-driving vehicles [34], which could be adapted for a
set of assertions covering swarm behaviour.
Runtime verification techniques can also be used as a post-design
strategy to build assurance, where an oracle (a system capable of
predicting or generating anticipated outcomes) is used to compare
current and expected behaviour [54]. Runtime verification requires
a suitable oracle to be defined which sets the expected bounds of
operation. Such a technique can be especially useful for complex
systems which are expected to change regularly and require regular
software updates which occur without time for a complete system
re-verification.
Taken together, verification of swarm robustness, scalability
and adaptability are suited to a mix of formal and testing-based
verification. A Probabilistic Finite State Machine (PFSM) verification
approach, in particular, could be suitable to prove system robustness
in terms of network connectivity and task completion rate [81].
Complementary to this, simulation should be used for assertion-
based analysis of the system in order to ensure validation and
specifications are met, especially when it comes to investigating
edge cases. Additionally, simulation may be used for capturing
scalability information by increasing the swarm size, and reliability
information compiled from long horizon observations. However, if
long-horizon testing is not feasible, but high reliability is needed for
low probability faults (such as safety critical aspects of the system),
then reliability models can be used to estimate future fault rates that
can be assessed against a specified fault tolerance level [16]. As a next
step, physical and hardware testing should be conducted in order
to validate formal and simulation-based methods and potentially
explore scenarios not possible in either of the non-physical methods.
As physical testing is usually the more costly method, a test plan
should set out and prioritise the most important trust properties to
analyse. In doing so, validation of simulated results will help to build
a strong and corroborative argument of swarm trustworthiness.
6 CONCLUSION
As robot swarms leave the laboratory to enter real world applica-
tions, swarm engineers will need to design these systems in safe
and useful ways that allow humans to trust them. However, devel-
oping trustworthy swarms is fraught with many challenges. For
example, the emergent behaviour of a swarm may come in at spo-
radic and unpredictable times, with changing behaviour across
different environments. In order to address concerns about the un-
predictability of a swarm system, we have highlighted a variety of
areas researching the “properties” of swarms as a means to produce
information and build trust in their behaviour. These properties
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
can be tested by a suite of both existing and emerging formal speci-
fication, verification and validation methods as a means of building
assurance. Such techniques, along with emerging best practice in
Human-Swarm Interactions research, may in turn incorporate the
human element of trustworthiness. Going forward, innovative ap-
proaches that consider both technical and social considerations
(such as human readable interfaces and visualisations) are essential
if we are to build trust in swarms before deployment.
REFERENCES
[1] 2012. A taxonomy ofmodel-based testing approaches. Software Testing, Verifica-
tion and Reliability Volume 22, 5 (2012), 297–312.
[2] 2015 - 2014. Big Hero 6 (Disney DVD). Disney, California.
[3] Dhaminda B. Abeywickrama, Amel Bennaceur, Greg Chance, Yiannis Demiris,
Anastasia Kordoni, Mark Levine, Luke Moffat, Luc Moreau, Mohammad Reza
Mousavi, Bashar Nuseibeh, Subramanian Ramamoorthy, Jan Oliver Ringert, James
Wilson, Shane Windsor, and Kerstin Eder. 2022. On Specifying for Trustworthi-
ness. (2022). arXiv:2206.11421
[4] Dhaminda B. Abeywickrama, James Wilson, Suet Lee, Greg Chance, Peter D.
Winter, Arianna Manzini, Ibrahim Habli, Shane Windsor, Sabine Hauert, and
Kerstin Eder. 2023. AERoS: Assurance of Emergent Behaviour in Autonomous
Robotic Swarms. arXiv:2302.10292 [cs.RO]
[5] Sander Ackermans, Debargha Dey, Peter Ruijten, Raymond H Cuijpers, and
Bastian Pfleging. 2020. The effects of explicit intention communication, conspic-
uous sensors, and pedestrian attitude in interactions with automated vehicles.
In Proceedings of the 2020 chi conference on human factors in computing systems.
1–14.
[6] HLEG AI. 2019. High-level expert group on artificial intelligence. Ethics guidelines
for trustworthy AI, 6.
[7] C Kristopher Alder, Samuel J McDonald, Mark B Colton, and Michael A Goodrich.
2015. Toward haptic-based management of small swarms in cordon and patrol.
In 2015 Swarm/Human Blended Intelligence Workshop (SHBI). IEEE, 1–8.
[8] Merihan Alhafnawi, Edmund R. Hunt, Severin Lemaignan, Paul O’Dowd, and
Sabine Hauert. 2022. Deliberative Democracy with Robot Swarms. In 2022
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 7296–
7303.
[9] Merihan Alhafnawi, Edmund R. Hunt, Severin Lemaignan, Paul O’Dowd, and
Sabine Hauert. 2022. MOSAIX: a Swarm of Robot Tiles for Social Human-Swarm
Interaction. In 2022 International Conference on Robotics and Automation (ICRA).
6882–6888.
[10] Victoria Alonso and Paloma De La Puente. 2018. System transparency in shared
autonomy: A mini review. Frontiers in neurorobotics 12 (2018), 83.
[11] Dan Amir and Ofra Amir. 2018. Highlights: Summarizing agent behavior to
people. In Proceedings of the 17th International Conference on Autonomous Agents
and MultiAgent Systems. 1168–1176.
[12] Jan Dyre Bjerknes and Alan FT Winfield. 2013. On fault tolerance and scalability
of swarm robotic systems. In Distributed autonomous robotic systems. Springer,
431–444.
[13] Josh C Bongard. 2013. Evolutionary robotics. Commun. ACM 56, 8 (2013), 74–83.
[14] Manuele Brambilla, Eliseo Ferrante, Mauro Birattari, and Marco Dorigo. 2013.
Swarm robotics: a review from the swarm engineering perspective. Swarm
Intelligence 7, 1 (2013), 1–41.
[15] Daniel S Brown, Michael A Goodrich, Shin-Young Jung, and Sean Kerman. 2016.
Two invariants of human-swarm interaction. Technical Report. AIR FORCE
RESEARCH LAB ROME NY ROME United States.
[16] Ricky W. Butler and George B. Finelli. 1993. The Infeasibility of Quantifying the
Reliability of Life-Critical Real-Time Software. IEEE Transactions on Software
Engineering 19, 1 (1993), 3–12.
[17] S Camazine, JL Deneubourg, NR Franks, J Sneyd, G Theraulaz, and E Bonabeau.
2001. Self-Organization in Biological Systems. (2001).
[18] Daniel Carrillo-Zapata, Emma Milner, Julian Hird, Georgios Tzoumas, Paul J
Vardanega, Mahesh Sooriyabandara, Manuel Giuliani, Alan FT Winfield, and
Sabine Hauert. 2020. Mutual shaping in swarm robotics: User studies in fire and
rescue, storage organization, and bridge inspection. Frontiers in Robotics and AI
7 (2020), 53.
[19] Greg Chance, Abanoub Ghobrial, Severin Lemaignan, Tony Pipe, and Kerstin
Eder. 2020. An agency-directed approach to test generation for simulation-
based autonomous vehicle verification. In 2020 IEEE International Conference On
Artificial Intelligence Testing (AITest). IEEE, 31–38.
[20] Mingxuan Chen, Ping Zhang, Xiaodan Chen, Yuliang Zhou, Fang Li, and Guan-
glong Du. 2018. A Human-swarm Interaction Method Based on Augmented
Reality. In 2018 WRC Symposium on Advanced Robotics and Automation (WRC
SARA). IEEE, 108–114.
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
[21] Erin K. Chiou and John D. Lee. 2021. Trusting Automation: Designing for Re-
sponsivity and Resilience. Human Factors (2021).
[22] Anders Lyhne Christensen, Rehan OGrady, and Marco Dorigo. 2009. From
Fireflies to Fault-Tolerant Swarms of Robots. IEEE Transactions on Evolutionary
Computation 13, 4 (2009), 754–766.
[23] Michele Colledanchise and Petter Ögren. 2018. Behavior trees in robotics and AI:
An introduction. CRC Press.
[24] Mohammad Divband Soorati, Jediah Clark, Javad Ghofrani, Danesh Tarapore, and
Sarvapali D Ramchurn. 2021. Designing a User-Centered Interaction Interface
for Human–Swarm Teaming. Drones 5, 4 (2021), 131.
[25] Marco Dorigo, Guy Theraulaz, and Vito Trianni. 2021. Swarm Robotics: Past,
Present, and Future [Point of View]. Proc. IEEE 109, 7 (2021), 1152–1165.
[26] Madeleine Clare Elish. 2019. Moral crumple zones: Cautionary tales in human-
robot interaction (pre-print). Engaging Science, Technology, and Society (pre-print)
(2019).
[27] Sondre A Engebråten, Jonas Moen, Oleg Yakimenko, and Kyrre Glette. 2018.
Evolving a repertoire of controllers for a multi-function swarm. In International
Conference on the Applications of Evolutionary Computation. Springer, 734–749.
[28] Luciano Floridi. 2019. Establishing the rules for building trustworthy AI. Nature
Machine Intelligence 1, 6 (2019), 261–262.
[29] Luciano Floridi, Josh Cowls, Monica Beltrametti, Raja Chatila, Patrice Chazerand,
Virginia Dignum, Christoph Luetge, Robert Madelin, Ugo Pagallo, Francesca
Rossi, Burkhard Schafer, Peggy Valcke, and Effy Vayena. 2018. AI4People—An
Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and
Recommendations. Minds and Machines 28, 4 (2018), 689–707.
[30] Olivier Graham Miller and Vaibhav Gandhi. 2021. A survey of modern exogenous
fault detection and diagnosis methods for swarm robotics. Journal of King Saud
University - Engineering Sciences 33, 1 (2021), 43–53.
[31] Azra Habibovic, Victor Malmsten Lundgren, Jonas Andersson, Maria Klingegård,
Tobias Lagström, Anna Sirkka, Johan Fagerlönn, Claes Edgren, Rikard Fredriks-
son, Stas Krupenia, et al. 2018. Communicating intent of automated vehicles to
pedestrians. Frontiers in psychology (2018), 1336.
[32] Heiko Hamann and Heinz Worn. 2007. A space-and time-continuous model of
self-organizing robot swarms for design support. In First International Conference
on Self-Adaptive and Self-Organizing Systems (SASO 2007). IEEE, 23–23.
[33] Peter A Hancock, Deborah R Billings, Kristin E Schaefer, Jessie YC Chen, Ewart J
De Visser, and Raja Parasuraman. 2011. A meta-analysis of factors affecting trust
in human-robot interaction. Human factors 53, 5 (2011), 517–527.
[34] Christopher Harper, Greg Chance, Abanoub Ghobrial, Saquib Alam, Tony Pipe,
and Kerstin Eder. 2021. Safety Validation of Autonomous Vehicles using Assertion-
based Oracles. arXiv preprint arXiv:2111.04611 (2021).
[35] Elliott Hogg, Sabine Hauert, David Harvey, and Arthur Richards. 2020. Evolving
behaviour trees for supervisory control of robot swarms. Artificial Life and
Robotics 25, 4 (2020), 569–577.
[36] Elliott Hogg, Sabine Hauert, and Arthur G Richards. 2021. Evolving Robust
Supervisors for Robot Swarms in Uncertain Complex Environments. DARS-
SWARM 2021 : Joint conference between distributed autonomous robotic systems
(DARS) and SWARM robotics conference ; Conference date: 01-06-2021 Through
04-06-2021.
[37] Edmund R. Hunt and Sabine Hauert. 2020. A checklist for safe robot swarms.
Nature Machine Intelligence 2 (07 2020).
[38] Edmund R Hunt, Simon W Jones, and Sabine Hauert. 2019. Testing the limits of
pheromone stigmergy in high-density robot swarms. Royal Society Open Science
6, 11 (6 Nov. 2019).
[39] Edmund R Hunt. 2020. Phenotypic plasticity provides a bioinspiration framework
for minimal field swarm robotics. Frontiers in Robotics and AI 7 (2020), 23.
[40] Aya Hussein and Hussein Abbass. 2018. Mixed initiative systems for human-
swarm interaction: Opportunities and challenges. In 2018 2nd Annual Systems
Modelling Conference (SMC). IEEE, 1–8.
[41] Aya Hussein, Leo Ghignone, Tung Nguyen, Nima Salimi, Hung Nguyen, Min
Wang, and Hussein A Abbass. 2018. Towards bi-directional communication in
human-swarm teaming: A survey. arXiv preprint arXiv:1803.03093 (2018).
[42] Amelia Ritahani Ismail, Jan Bjerknes, Jon Timmis, and Alan Winfield. 2015. An
Artificial Immune System for Self-Healing in Swarm Robotic Systems. 61–74.
[43] Simon Jones, Matthew Studley, Sabine Hauert, and Alan Winfield. 2018. Evolving
behaviour trees for swarm robotics. In Distributed Autonomous Robotic Systems.
Springer, 487–501.
[44] Simon Jones, Alan F Winfield, Sabine Hauert, and Matthew Studley. 2019. On-
board evolution of understandable swarm behaviors. Advanced Intelligent Systems
1, 6 (2019), 1900031.
[45] Lawrence H Kim, Daniel S Drew, Veronika Domova, and Sean Follmer. 2020.
User-defined swarm robot control. In Proceedings of the 2020 CHI Conference on
Human Factors in Computing Systems. 1–13.
[46] Bing Cai Kok and Harold Soh. 2020. Trust in robots: Challenges and opportunities.
Current Robotics Reports 1, 4 (2020), 297–309.
[47] Andreas Kolling, Phillip Walker, Nilanjan Chakraborty, Katia Sycara, and Michael
Lewis. 2015. Human interaction with robot swarms: A survey. IEEE Transactions
on Human-Machine Systems 46, 1 (2015), 9–26.
Sabine Hauert et al.
[48] John Koza. 1994. Evolution of emergent cooperative behavior using genetic
programming. Computing with Biological Metaphors (1994), 280–297.
[49] Tobias Lagström and Victor Malmsten Lundgren. 2016. AVIP-Autonomous vehicles’
interaction with pedestrians-An investigation of pedestrian-driver communication
and development of a vehicle external interface. Master’s thesis.
[50] John D. Lee and Katrina A. See. 2004. Trust in automation: Designing for appro-
priate reliance. Human Factors 46, 1 (2004), 50–80.
[51] John D Lee and Bobbie D Seppelt. 2009. Human factors in automation design. In
Springer handbook of automation. Springer, 417–436.
[52] Suet Lee, Emma Milner, and Sabine Hauert. 2022. A Data-Driven Method for
Metric Extraction to Detect Faults in Robot Swarms. IEEE Robotics and Automation
Letters 7, 4 (2022), 10746–10753.
[53] Joel Lehman and Kenneth O Stanley. 2011. Evolving a diversity of virtual creatures
through novelty search and local competition. In Proceedings of the 13th annual
conference on Genetic and evolutionary computation. 211–218.
[54] Martin Leucker and Christian Schallhart. 2009. A brief account of runtime
verification. Journal of Logic and Algebraic Programming 78, 5 (2009), 293–303.
[55] Michael Lewis. 2013. Human interaction with multiple remote robots. Reviews of
Human Factors and Ergonomics 9, 1 (2013), 131–174.
[56] Sebastian Lindner and Axel Schulte. 2021. Evaluation of Swarm Supervision
Complexity. In International Conference on Intelligent Human Systems Integration.
Springer, 50–55.
[57] Carl Macrae. 2022. Learning from the failure of autonomous and intelligent
systems: Accidents, safety, and sociotechnical sources of risk. Risk analysis 42, 9
(2022), 1999–2025.
[58] Karthik Mahadevan, Sowmya Somanath, and Ehud Sharlin. 2018. Communi-
cating awareness and intent in autonomous vehicle-pedestrian interaction. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
1–12.
[59] Clark A Miller and Ira Bennett. 2008. Thinking longer term about technology:
is there value in science fiction-inspired approaches to constructing futures?
Science and Public Policy 35, 8 (2008), 597–606.
[60] Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping
elites. arXiv preprint arXiv:1504.04909 (2015).
[61] Changjoo Nam, Huao Li, Shen Li, Michael Lewis, and Katia Sycara. 2018. Trust of
humans in supervisory control of swarm robots with varied levels of autonomy.
In 2018 ieee international conference on systems, man, and cybernetics (smc). IEEE,
825–830.
[62] Steven Nunnally, P Walker, Andreas Kolling, Nilanjan Chakraborty, Michael
Lewis, K Sycara, and M Goodrich. 2012. Human influence of robotic swarms
with bandwidth and localization issues. In 2012 IEEE International Conference on
Systems, Man, and Cybernetics (SMC). IEEE, 333–338.
[63] James O’Keeffe, Danesh Tarapore, Alan G. Millard, and Jonathan Timmis. 2018.
Adaptive online fault diagnosis in autonomous robot swarms. Frontiers in Robotics
and AI 5 (November 2018).
[64] Jayam Patel, Yicong Xu, and Carlo Pinciroli. 2019. Mixed-granularity human-
swarm interaction. In 2019 International Conference on Robotics and Automation
(ICRA). IEEE, 1059–1065.
[65] Jacques Penders, Lyuba Alboul, Ulf Witkowski, Amir Naghsh, Joan saez pons,
Stefan Herbrechtsmeier, and Mohamed Habbal. 2011. A Robot Swarm Assisting
a Human Fire-Fighter. Advanced Robotics 25 (01 2011), 93–117.
[66] Thomas Schmickl and Karl Crailsheim. 2006. Trophallaxis among swarm-robots:
A biologically inspired strategy for swarm robotics. In The First IEEE/RAS-EMBS
International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob
2006. IEEE, 377–382.
[67] Melanie Schranz, Martina Umlauft, Micha Sende, and Wilfried Elmenreich. 2020.
Swarm Robotic Behaviors and Current Applications. Frontiers in Robotics and AI
7 (2020).
[68] Mohammad Divband Soorati, Javad Ghofrani, Payam Zahadat, and Heiko
Hamann. 2018. Robust and adaptive robot self-assembly based on vascular
morphogenesis. In 2018 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS). IEEE, 4282–4287.
[69] Daniel P Stormont. 2008. Analyzing human trust of autonomous systems in
hazardous environments. In Proc. of the Human Implications of Human-Robot
Interaction workshop at AAAI. 27–32.
[70] Volker Strobel, Eduardo Castelló Ferrer, and Marco Dorigo. 2020. Blockchain
Technology Secures Robot Swarms: A Comparison of Consensus Protocols and
Their Resilience to Byzantine Robots. Frontiers in Robotics and AI 7 (2020).
[71] Danesh Tarapore, Pedro Lima, Jorge Carneiro, and Anders Christensen. 2015. To
err is robotic, to tolerate immunological: Fault detection in multirobot systems.
Bioinspiration and biomimetics 10 (02 2015), 016014.
[72] UK Driving Standards Agency. 2012. The Official Highway Code. Her Majestys
Stationery Office.
[73] Emil Vassev, Roy Sterritt, Christopher Rouff, and Mike Hinchey. 2012. Swarm
Technology at NASA: Building Resilient Systems. IT Professional 14, 2 (2012),
36–42.
[74] Phillip Walker, Steven Nunnally, Mike Lewis, Andreas Kolling, Nilanjan
Chakraborty, and Katia Sycara. 2012. Neglect benevolence in human control
Trustworthy Swarms
of swarms in the presence of latency. In 2012 IEEE International Conference on
Systems, Man, and Cybernetics (SMC). IEEE, 3009–3014.
[75] Toby Walsh. 2017. Android Dreams: The past, present and future of artificial
intelligence. Oxford University Press.
[76] Ning Wang, David V Pynadath, and Susan G Hill. 2016. Trust calibration within
a human-robot team: Comparing automatically generated explanations. In 2016
11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE,
109–116.
[77] Stop Autonomous Weapons. 2017. Slaughterbots. YouTube. https://www.youtube.
com/watch?v=9CO6M2HsoIA
[78] Justin Werfel, Kirstin Petersen, and Radhika Nagpal. 2014. Designing Collective
Behavior in a Termite-Inspired Robot Construction Team. Science (New York,
TAS ’23, July 11–12, 2023, Edinburgh, United Kingdom
N.Y.) 343 (02 2014), 754–8.
[79] James Wilson. 2022. Search Space Illumination of Robot Swarm Parameters for
Trustworthiness. In Swarm Intelligence: 13th International Conference, ANTS 2022,
Málaga, Spain, November 2–4, 2022, Proceedings, Vol. 13491. Springer Nature, 378.
[80] James Wilson, Jon Timmis, and Andy Tyrrell. 2018. A Hormone-Inspired Arbi-
tration System For Self Identifying Abilities Amongst A Heterogeneous Robot
Swarm. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE,
843–850.
[81] Alan F.T. Winfield, Wenguo Liu, Julien Nembrini, and Alcherio Martinoli. 2008.
Modelling a wireless connected swarm of mobile robots. Swarm Intelligence 2,
2-4 (2008), 241–266.
[82] Jeannette M. Wing. 2021. Trustworthy AI. Commun. ACM 64, 10 (2021), 64–71.