Available online at www.sciencedirect.

com

ScienceDirect
IFAC PapersOnLine 55-2 (2022) 295–300

Digital
Digital Twin-based
Twin-based Design
Design and
and Operation
Operation of of Human-Robot
Human-Robot Collaborative
Collaborative
Digital Twin-based Design and Operation
Assembly of Human-Robot Collaborative
Assembly
Assembly of Human-Robot Collaborative
Digital Twin-based Design and Operation
Yichen
Yichen Wang*,
Wang*, Jindan
Jindan Feng**,
Feng**, Jinshan Liu**,
Liu**, Xiaojun
Assembly
Jinshan Xiaojun Liu***,
Liu***, Junfeng
Junfeng Wang*
Wang*
Yichen
Yichen Wang*,
Wang*, Jindan
Jindan Feng**,
Feng**, Jinshan
Jinshan Liu**,
Liu**, Xiaojun
Xiaojun Liu***,
Liu***, Junfeng
Junfeng Wang*
Wang*
Yichen Wang*, Jindan Feng**, Jinshan Liu**, Xiaojun Liu***, Junfeng Wang*
*Department
*Department of of Industrial
of Industrial
Industrial and and Manufacturing
and Manufacturing
Manufacturing System System Engineering,
System Engineering,
Engineering, Huazhong Huazhong
Huazhong UniversityUniversity
University of of Science
of Science
Science and and
and
*Department
*Department Yichen Wang*,
of Industrial
Industrial Jindan Feng**,
and Manufacturing
Manufacturing Jinshan
System Liu**, Xiaojun
Engineering, Liu***,
Huazhong Junfeng
University Wang*
of Science
Science and and
*Department Technology,
of
Technology, Wuhan,
and
Wuhan, 430074
430074 China
China (corresponding
System Engineering,
(corresponding e-mail:
e-mail: wangjf@hust.edu.cn).
Huazhong University
wangjf@hust.edu.cn). of
Technology,
Technology, Wuhan,
Wuhan, 430074
430074 China
China (corresponding
(corresponding e-mail:
e-mail: wangjf@hust.edu.cn).
wangjf@hust.edu.cn).
Technology,
*Department of Industrial Wuhan,
**Beijing 430074
andSpacecrafts
Manufacturing China (corresponding
Limited
SystemCompany, e-mail:
Beijing,
Engineering, wangjf@hust.edu.cn).
100094
Huazhong China
University of Science and
**Beijing
**Beijing Spacecrafts
Spacecrafts Limited
Limited Company,
Company, Beijing, 100094 China China
Technology, **Beijing
Wuhan,
**Beijing Spacecrafts
430074 China
Spacecrafts Limited Company, Beijing,
(corresponding
Limited Company, Beijing, 100094
100094
e-mail:100094
Beijing, China
wangjf@hust.edu.cn).
China
*** School
*** School
School of of Mechanical
of Mechanical Engineering,
Mechanical Engineering, Southeast
Engineering, Southeast University,
Southeast University,
University, Nanjing, Nanjing, 211198
Nanjing, 211198 China
China
***
*** 211198 China
*** School of Mechanical Engineering, Southeast University, Nanjing, 211198 China
School of Mechanical
**Beijing SpacecraftsEngineering,
Limited Southeast
Company, University,
Beijing, Nanjing,
100094 211198
China China
Abstract:
Abstract: Human-robot
Human-robot
*** collaborative
collaborative
School of Mechanical assembly
assembly
Engineering, (HRCA)
(HRCA)
Southeast can
can reduce
reduce
University, human
human
Nanjing, work
work intensity
intensity
211198 China and
and improve
improve
Abstract:
Abstract: Human-robot
Human-robot collaborative
collaborative assembly
assembly (HRCA)
(HRCA) can reduce
can reduce
reduce human
human work
work not intensity
intensity and
and improve
improve
the
the level
Abstract:
level of
of assembly
Human-robot
assembly automation.
collaborative
automation. However,
assembly
However, high
(HRCA)
high assembly
can
assembly efficiency
efficiency human requires
work
requires not only
intensity
only aaandreasonable
improve
reasonable
the
the level
level of
of assembly
assembly automation.
automation. However,
However, high
high assembly
assembly efficiency
efficiency requires
requires not
not only
only aa reasonable
reasonable
division
the levelof
division of
ofoflabor
assembly
labor between
between humans
automation.
humans and and
and robots,
However, but
robots, high
but also close
assembly
also close communication
efficiency
communication requiresand flexible
not
and flexible
flexible control
onlycontrol
aand
control during
reasonable
during
Abstract:
division Human-robot
labor between collaborative
humans assembly
robots, but(HRCA)
also can communication
close reduce human work
and intensity improve
during
division
the assembly
division
the assembly of
of labor
labor between
process.
between
process. In
In humans
this paper,
humans
this paper, and
we
and
we robots,
propose
robots,
propose but
a
buta also
Digital
also
Digitalclose
Twin
close
Twin communication
(DT)
communication
(DT) based
based and
framework
and
framework flexible
flexible for
for control
the
control
the design
designduring
and
during
and
the level of assembly
assembly process. Inautomation.
this paper, However,
we propose high assembly
aa Digital Twin efficiency
(DT) based requires
framework not onlyfor the a design
reasonable and
the
the assembly
operation
assembly
operation of
of process.
HRCA.
process.
HRCA. In
The
In
The this
this paper,
framework
paper,
framework we
we propose
consists
propose
consists of
of five
a
fiveDigital
Digital Twin
elements:
Twin
elements: (DT)
physical
(DT)
physical based
based framework
entities, virtual
framework
entities, virtual for
for the
entities,
the
entities, design
twin
design
twin and
data,
and
data,
division ofoflabor
operation HRCA. between
The humans and
framework robots,ofbut
consists five also close communication
elements: physical entities, and flexible
virtual control
entities, twin during
data,
operation
service
operation
service of
centers HRCA.
of HRCA.
centers and
and The
their
The
their framework
connections.
framework
connections. consists
In
consists
In of five
particular,
of five
particular, elements:
the
elements:
the physical
design
physical
design process
process entities,
is
entities,
is virtual
oriented
oriented entities,
to
virtual entities,
to layout twin
twin data,
design,
data,
the assembly
service
service centers
centers
process.
and
and
In this
their
their
paper, we
connections.
connections.
propose
In
In
a Digital
particular,
particular, the
the
Twin
design
design
(DT) based
process framework
is oriented to layout
for the design
layout design,
and
design,
sequencecenters
service
sequence planning
planning and and
their
and task
task assignment;
connections.
assignment; In the operation
operation
particular,
the the process process
design
process is oriented
process
is is
oriented
is oriented
oriented
to datato
to data to layout
processing
layout
processing design,
and
design,
and
operation
sequence of HRCA.
planning The
and framework
task assignment;consists of
the five elements:
operation physical
process is entities,
oriented virtual
to data entities,
processing twin data,
and
sequence
visualization,
sequence planning
planningevent and task assignment;
processing, task scheduling the
the operation
scheduling process
and human-robot is oriented
control. to
to data
Through processing
virtual and reality and
reality
visualization,
service
visualization,centers and and
event
event theirtask
processing, assignment;
task
connections.
processing, task operation
In particular,
scheduling and process process
and human-robot
the design
human-robot is oriented
control.
control. isThrough
oriented
Through data to processing
virtual
virtuallayout and
and and
design,
reality
visualization,
real-time mapping,
visualization,
real-time mapping, event
event processing,
simulation
processing,
simulation and
andtask
task scheduling
optimization,
scheduling
optimization, itand
and
it human-robot
enables
human-robot
enables deep control.
deep human-robot
human-robot
control. Through
collaboration
Through
collaborationvirtual
virtual and
and
and
and reality
efficient
reality
efficient
sequence mapping,
real-time planning simulation
and task assignment;
and optimization, the operation
it enables process
deep is orientedcollaboration
human-robot to data processing and and
efficient
real-time
assembly.mapping,
assembly.
real-time We
mapping, finallysimulation
use
use the
simulation the and
and optimization,
example of it
of satellite
optimization, enables
itand plate
enables deep
assembly
deep human-robot
to
to illustrate
human-robot collaboration
the
collaboration and
rationality
and efficient
of
of the
visualization,
assembly. Weevent
We finally
finally processing,
use the example
task
example scheduling
of satellite
satellite plate
plate assembly
human-robot
assembly control.
to illustrate
Through
illustrate the virtual
the rationality
rationality andefficient the
reality
of the
assembly. We
framework
assembly.
framework We
in
in the
thefinally
designuse
finally
design use
andthe
and the example
operation
example
operation of of satellite
of HRCA.
HRCA.
of satellite plate
plate assembly
assembly to to illustrate
illustrate the the rationality
rationality of
of the
the
real-time
framework mapping,
in the simulation
design and and optimization,
operation of HRCA. it enables deep human-robot collaboration and efficient
framework
framework
Keywords:
Keywords:
Copyright ©in
in the
Human-robot
the
Human-robot
2022 design
design
The and
and operation
collaboration,
operation
collaboration,
Authors. This is of
of
an HRCA.
Assembly,
HRCA.
Assembly,
open access Digital
article
Digital Twin,
under
Twin, the Design
CC
Design and
BY-NC-ND
and Operation
license
Operation
assembly. We
Keywords: finally usecollaboration,
Human-robot the example Assembly, of satellite Digital
plate assembly
Twin, Design to illustrate
and Operationthe rationality of the
Keywords: Human-robot
(https://creativecommons.org
Keywords:
framework Human-robot
in the design and collaboration, Assembly,
Assembly, Digital
/licenses/by-nc-nd/4.0/)
collaboration,
operation of HRCA. Digital Twin,
Twin, Design Design and and Operation
Operation
Keywords: Human-robot collaboration, Assembly, Digital Twin, Design and Operation
The
The rest
rest of of thisthis paper
paper is
is organized
organized as as follows.
follows. SectionSection 2
2
1. INTRODUCTION
INTRODUCTION The
1.
1. INTRODUCTION The rest
The rest of
summarizes
rest
summarizes of the
of
this
this
the
paper
thisrelevant
paper
relevant
is
is organized
paper studies
is organized
on
organized
studies on HRCA
HRCA
as
as follows.
as follows.
and
follows.
and DT.
DT.
Section
Section
Section
Section
Section
2
2
3
2
3
1.
1. INTRODUCTION
INTRODUCTION summarizes the relevant studies on HRCA and DT. Section 3
Assembly
Assembly is
is an
an important
important process
process from
from parts
parts to
to product,
product, and
and summarizes
proposes
summarizes
proposes the relevant
aa DT-based
the
DT-basedrelevant studiesframework
overall
studies
overall on HRCA
on HRCAfor
framework andthe
and
for DT.design
DT.
the Section
Section
design and
and3
3
The rest aofDT-based
proposes this paperoverall is organized
framework as follows.
for Sectionand2
Assembly
Assembly
automation
is
ishasan
an animportant
important
important
process
process
1. INTRODUCTION impact
from
from
on
parts
parts to
improving to product,
product,
production
and proposes
operation aaprocess
and proposes DT-based
process
DT-based overallSection
of HRCA.
HRCA.
overall framework
Section
framework for the
4 describes
describes
for the thedesign
design
the and
detailed
Assembly
automation
automation ishas
hasan important
an
an important
important process
impact
impact from
on
on parts
improving
improving to product,
production
production and operation
summarizes
operation of
the relevant
process of HRCA. studies on HRCA
Section 4
4 andthe
describes DT. design
the Section and
detailed
detailed 3
automation However,
efficiency.
automation
efficiency. has an
has
However, an important
important
many
many impact on
products
impact
products on
have
haveimproving
complex
improving
complex production
structures
production
structures operation
sub-framework
operation
sub-framework process
process of
of of
the
of
the HRCA.
two
HRCA.
two Section
processes,
Section
processes, 4
4 describes
which includes
describes
which includes the
the detailed
various
detailed
various
Assembly isHowever,
efficiency. an important many process
products fromhave parts
complexto product, and proposes
structures sub-framework
sub-framework
a DT-based of
of the
the
overall
two
two
framework
processes, which forincludes
the design and
various
efficiency.
and
and aa large
efficiency.
aa large
automation
and large
However,
number
However,
number
has
number
of
ofmany
an important
of
parts
many
parts
parts
products
which
products
which
impact
which
have
require
have
require
on
complex
aa high
complex
aa high
improving
require high
structures
degree
structures
degree
production
degree
of
of
of
services.
sub-framework
services.
operation
services.
Section
Section
process
Section of5 5
5 the
of two processes,
verifies
verifies
HRCA.
verifies
the
processes,
the
Section
the
whichand
rationality
4which
rationality
rationality
includes
includes
and
describes
and
various
feasibility
various
feasibility
the detailed
feasibility
of
of
of
and
flexibility
and a
flexibilitylarge
large in
innumber
the
number
the of
assembly
of
assembly parts
parts which
process.
which
process. require
Traditional
require a
Traditional high
high degree
robots
degree
robots of
are
of
are services.
the
the framework
services.
frameworkSection
Section 5
5 verifies
through
verifies
through a
a the
case
the
case rationality
study
rationality
study of
of and
satellite
and
satellitefeasibility
assembly.
feasibility
assembly. of
of
efficiency. in
flexibility However,
the many products
assembly process. have complexrobots
Traditional structuresare sub-framework
the framework of the twoa processes,
through case study which
of includes
satellite various
assembly.
flexibility
unable
flexibility
unable to
to in
assemble
in
assemblethe
the assembly
such
assembly
such process.
products
process.
products due
due Traditional
to their
Traditional
to their large
large robots
size
robots
size are
and
are
and the
the framework
Finally, Section
framework
Finally, Section 6
6through
gives
through
gives a
a a
a case
conclusion.
case
conclusion. study
study of
of satellite
satellite assembly.
assembly.
and a large
unable to number such
assemble of parts whichdue
products require
to theira high
large degree
size and services.Section
of Finally, Section6 5gives verifies the rationality and feasibility of
a conclusion.
unable
bulky
unable
bulky to
to assemble
movements.
assemble
movements. such
With
such
With products
the
products
the due
development to
to their
theirof
due Traditional
development oflarge size
sensor
large size and
sensor and Finally,
Finally, Section
Section 6 gives
gives aa aconclusion.
6through conclusion.
flexibility
bulky in the assembly
movements. With theprocess.
development of robots
sensor are the
and framework 2.
2. STATE
STATE case OF
OFstudy
ART
ART of satellite assembly.
bulky movements.
communication
bulky With
technologies, the
the development
more flexible of sensor
and and
safer
unable movements.
communication
to assemble
communication Withproducts
technologies,
such
technologies, development
more
more theiroflarge
due toflexible
flexible sensor
and
and and
sizesafer
and Finally, Section 6 gives
safer 2.
2.
2.
STATE
a conclusion.
STATE
STATE
OF
OF
OF
ART
ART
ART
communication
collaborative robots
communication
collaborative robots technologies,
(also known
technologies,
(also known more more
as cobots)
as flexible
cobots)
flexible
have and safer
have emerged
emerged
and saferto Human-robot
to Human-robot collaborationcollaboration is
is one
one of
of the
the closest
closest forms
forms of
of
bulky movements.
collaborative robots With known the development of sensor and
collaborative
provide
collaborative
provide robots (also
solutions
robots
solutions (also
to known
(also
to known
these as
these
as
as cobots)
cobots) have
problems.
cobots)
problems. have
have
emerged
emergedthe
Through
emerged
Through the
to
to Human-robot
to Human-robot
cooperation
Human-robot
cooperation
collaboration
collaboration
between 2.
collaboration
between humans
STATE
humans
is
is
is
one
one
and
OF
one
and
of
of
robots.
ART
of
robots.
the
the
the
closest
closest
The
The
forms
forms
characteristics
closest forms
characteristics
of
of
of
communication
provide solutions technologies,
to these more
problems. flexibleThrough and safer the cooperation between humans and robots. The characteristics
provide solutions
collaboration
provide
collaboration solutions
betweento
between tohumans
humans theseand
these problems.
andproblems.
robots, the
robots, Through
the Through
advantagesthe
advantages the
of of
of cooperation
of it is
cooperation
it is that
that between
H-R
between
H-R share
share humans
workspace,
humans
workspace,and
and robots.
tasks
robots.
tasks The
and
The
and characteristics
resources
characteristics
resources and
and
collaborative robots
collaboration between (also
humansknown andas robots,
cobots)the have emerged of
advantages to Human-robot
of it is that H-Rcollaboration
share workspace, is one of tasks theandclosest formsand
resources of
collaboration
creativity,
collaboration high between
between humans
flexibility, and
learning robots,
and the
cognitive advantages
ability of accomplish
of it
it isis that H-R
tasks share workspace, tasks
whileand resources and
creativity,
provide solutions
creativity, high
high tohumans
flexibility,
flexibility, these andproblems.
learning
learning robots,
and
and the Through
cognitive
cognitive advantages
ability
ability of
the
of accomplish
of
cooperation
accomplish that between
H-R
tasks
tasks
simultaneously
share workspace,
simultaneously
humans
simultaneously tasks
and robots.while
while and
The
allowing
allowing direct
resourcesdirect
characteristics
allowing and
direct
creativity,
humans,
creativity,
humans, and
andhigh
highthe
the flexibility,
strength,
flexibility,
strength, learning
fatigue-free,
learning
fatigue-free, and
and cognitive
speed and
cognitive
speed ability
accuracy
ability of of accomplish
contact (Wang tasks
et al., simultaneously
al.,simultaneously
2019). There There are while
are allowing
manyallowing
studies on ondirect
the
collaboration
humans, and between
the strength, humans and robots,
fatigue-free, speed theand
and accuracy
advantages
accuracy accomplish
of contact
of it is (Wang
contact that H-R
(Wang tasks
et
et share
al., 2019).
workspace,
2019). There while
tasks
are many
and studies
many resources
studies ondirect
the
and
the
humans,
robots
humans,
robots and
can
and
can be
be the
the strength,
fully utilized
strength,
fully utilized fatigue-free,
(Hashemi-Petroodi
fatigue-free,
(Hashemi-Petroodispeed
speed and
and et
etaccuracy
al.,
accuracy
al., 2019).
2019).of
of contact
design
contact
design (Wang
and
(Wang
and et
operation
et
operation al.,
al., 2019).
of
2019).
of HRCA,
HRCA, There
There are
where
are
where many
the
many
the studies
design
studies
design on
aspects
on
aspectsthe
the
creativity,
robots can high
be flexibility,
fully utilized learning and cognitive
(Hashemi-Petroodi et ability
al., 2019).of design
accomplish and tasks
operation simultaneously
of HRCA, while
where the allowing
design direct
aspects
robots
In this
robots
In this can
way,
can
way, be
be not
notfully
only
fully
only utilized
the
utilized
the (Hashemi-Petroodi
reliability of
(Hashemi-Petroodi
reliability of the
the assembly
assembly et
et al.,
al., 2019).
can
2019).
can be
be design
focus
design
focus on
onand
and operation
scene
operation
scene layout
layout of
of and
andHRCA,
HRCA,task
task where
assignment;
where
assignment; the
the design
the
design
the aspects
operation
aspects
operation
humans,
In this way, and notthe strength,
only the fatigue-free,
reliability of speed
the and accuracy
assembly can of focus
In
In this
ensured,
this can way,
but
way, not
also
not only
the
only the
the reliability
productivity can
reliability of
be
of the assembly
improved.
the assembly can be be contacton(Wang
focus on scene
scene
etlayout
al., 2019).
layout and
and
Thereassignment;
task
task
are many studies
assignment; the
the
on the
operation
the operation
ensured,
robots
ensured, but
but bealso
also the
fully
the productivity
utilized
productivity can be
(Hashemi-Petroodi
can be improved.
improved. et al., can
2019).be aspects
aspects
focus
design
aspects onfocus
focus
and
focusscene on H-R
on
operation
on
H-R
layout
H-R
control
of and
control
control
and
HRCA,
and
safety.
tasksafety.
and assignment;
where
safety. the design operation
aspects
ensured, but but also the the productivity
productivity can can be be improved.
improved.
ensured,
In
Closer way, also
this human-robot
Closer not only
human-robot the reliability
(H-R)
(H-R) relationship
relationship ofimproves
the assembly
improves can be aspects
productivity,
productivity,
aspects
focusscene
The on
scene
focus
focusscene
layout
on
on H-RH-R
layout
needs
control
control
and
to be
and
and
task
safety.
safety.
assignment;
created the operation
or reconfigured
reconfigured if the
the
Closer human-robot (H-R) relationship improves productivity, The
The scene layout
layout needs
needs to
to be
be created
created or
or reconfigured if
if the
ensured,
Closer
but also but also
human-robot
alsohuman-robot
brings many the productivity
(H-R)
many(H-R) problems, can
relationship be improved.
improves productivity,
e.g. unreasonable
unreasonable productivity,
H-R tasktask product
aspects focus on H-R control and safety. if
Closer
but
but also brings
brings many problems,
problems,relationship
e.g.
e.g. improves
unreasonable H-R
H-R task The
product
The
product
scene
scene needs
needs
needs
layout
layout to
to
to
needs
be
needs
be
be
to
updated be
to be or
updated
updated
created
or if
created
or if
if
the
the
the
or reconfigured
customer
orcustomer
reconfigured
customer has if the
special
has special
has the
special
but
but also brings
assignment,
also
assignment, bringspoor many
poormany problems, robustness,
collaboration
problems,
collaboration e.g. unreasonable
e.g. unreasonable
robustness, and higher
and H-R H-R
higher
H-R task requirements.
H-R
task product
product
requirements.needs
needs to
Pooja
to
Pooja be
be updated
et al.
updated
et al. or
(2018)
or
(2018) if
if the
proposed
the
proposedcustomer
customer has special
aa collaborative
has special
collaborative
Closer
assignment, human-robot
poor (H-R)
collaboration relationship
robustness,improves and productivity,
higher H-R if the
assignment,
risk. Digital
assignment, poor
twin
poormany collaboration
(DT) is
collaboration now robustness,
widely
robustness,used inand
and higher
manufacturing
higher H-R The scene layout
requirements.
requirements. Pooja
Pooja
needset
et
to be
al.
al. (2018)
al. (2018)
created or reconfigured
proposed
proposed aa collaborative
a collaborative
risk.
but
risk.alsoDigital
brings
Digital twin
twin (DT)
(DT) is now
problems,
is now widely used
e.g. unreasonable
widely used in
in H-R H-R
manufacturing
manufacturing task unit product
unit
layout
requirements.
unit layout
layoutneeds
design
design
designPooja
to be
approach
et
approach
updated
approach
based
(2018)
based
or if on
based
on risk
proposed
on risk
therisk
assessment
assessment
customer
assessment
of
of the
collaborative
has special
of the
risk.
system
risk.
system Digital
Digital
assignment,
system
twin
optimization
twin
poor (DT)
optimization
optimization
(DT) is
(Tsubasa
(Tsubasa
collaboration
(Tsubasa
now
is now widelywidely
et
et
et
al.,
al., used
2020).
2020).
robustness,
al., 2020).
in
used inand One
One
One
manufacturing
alternative
manufacturing H-R task.
alternative
higher
alternative unit layout
unit
task. layout
They
They design
first
design
first approach
designed
approach
designed the
the based
scenario
based
scenario on using
on risk assessment
risk
using assessment
aa 3D
3D of the
simulation
of
simulation the
the
requirements.
task. Pooja et al. (2018) proposed a collaborative
system
approach
system
approach optimization
is to
optimization
is to introduce
introduce (Tsubasa
(TsubasaDT into
is DTnowinto
et
et al.,
the
al.,
the 2020).
HRCA
2020).
used inOne
HRCA One
process.
process.alternative
Based
alternative
Based task. They
tool.
task. They
Then
They
Then first
first
first
they
designed
designed
evaluated
theydesigned
the
thethe
the
scenario
scenario
security
scenario
using
using
risks
aa 3D
3D
of
of the
simulation
simulation
layout
risk. Digital
approach
approach
on the DT
is
is
twin
to
to
(DT)
introduce
introduce
concept to
DT
DT
establish
widely
into
into a
the
the HRCA
HRCA
digital
manufacturing
process.
process.
mapping of
Based
Based
each
tool.
unit layout
tool. Then design
they evaluated
approach
evaluated the
based
the on using
security
security risks
riskrisks a 3D
assessment
of simulation
the
the layout
of the
layout
approach
on the
system
on the DTis to
optimization
DT introduce
concept
concept to
(Tsubasa
to DT
establish
establish into athe HRCA
digital
et aal.,digital process.
mapping
2020).mapping of
One alternative
ofBased
each
each tool.
through
tool.
through Then
Then they
physical
they
physical evaluated
tests
evaluated
tests and
and the
the security
quantitative
security
quantitative risks
risks of
calculations.
of
calculations. the
the layout
Finally,
layout
Finally,
on the DT concept to establish
virtualaa space,
digital mapping task.
through They first
physical designed
tests and the scenario
quantitative using a 3D
calculations. simulation
Finally,
entity
on the on
entity on work-cell
DTis concept
work-cell in
to the
in establish
the DTvirtual digital
space, the
mapping
the physicalof
physical each
space
each the
ofBased
space through
layout
through physical
was
physical tests
tests and
optimized and quantitative
based on
quantitativethe calculations.
the evaluation
calculations. Finally,
results.
approach
entity
entity
assembly
on to introduce
work-cell
on process
work-cellis
process
in
in
is in
the
the virtual
driven
into the
virtual
HRCA
space, the
space, simulation,
by real-time
real-time
process.
the physical
physical
simulation,
space
space the
analysis
the layout
tool.layout
Then was
wasthey optimized
evaluated
optimized based
the on
based onsecurity
the evaluation
risks ofresults.
evaluation theFinally,
results. layout
entity
assembly
on on work-cell
the DTprocess
assembly concept is the
driven
to establish
driven virtual
by
by space, the
a digitalsimulation,
real-time physical
mapping analysis space
analysis
of each Task the
the layout was
layoutphysical optimized
was optimized based
based on the evaluation
on the evaluation results.
results.
assembly
and process
decision making is driven
driven
of theby bymapping
real-time(Wang
mapping simulation,
(Wang et al., analysis
al.,analysis
2019). Task through assignment
Task assignment
assignment determines tests
determinesand
determines the quantitative
the specific
the specific
specific tasks calculations.
tasks and Finally,
sequences
assembly
and
entity
and on process
decision making
work-cell
decision making is inofthethe real-time
virtual space, simulation, et
the physical 2019).
space Task tasks andand sequences
sequences
and
This
and
This decision
paper
decision
paper making of
establishes
making
establishes ofaa the
of the mapping
mappingHRCA
DT-based
the mapping
DT-based
(Wang
(Wangframework
(Wang
HRCA
et
et al.,
et
frameworkal., 2019).
al., 2019).
and
2019).
and
the
thatlayout
Task
that was
assignment
each
assignment
each optimized
human/robot
human/robotdetermines
determines basedthe
needs
the
needs onspecific
totheperform.
specific
to evaluation
perform.tasks
tasks and
A
and
A results.
sequences
reasonable
sequences
reasonable
assembly
This paperprocess
establishesis driven a by real-time
DT-based HRCA simulation,
framework analysis
and that
that each
each human/robot
human/robot needs
needs to
to perform.
perform. A
A reasonable
reasonable
This
elaborates
This
elaboratespaper
paper its
itsestablishes
design
establishes
design and
and aaoperation
DT-based
DT-based
operation HRCA
process
HRCA
process in
in framework
detail.
framework
detail. and
and assignment
that each
assignment scheme
human/robot
scheme can
can give
needs
give full
to
full play
perform.
play to
to the
A
the respective
reasonable
respective
and decision
elaborates its makingand ofoperation
the mapping (Wang et al., 2019). Task assignment
assignment scheme determines
can give the full
give specificplaytasks to theand respective
the sequences
elaborates
elaborates its design
design
design and and aoperation
process
process in
in detail.
in detail. assignment
advantages
assignment
advantages ofscheme
of human can
human
scheme and
can robot full
robot
give
and needs which
full
which play
play reduce
reduceto
to theassembly
assemblyrespective
time
respective
time
This paperitsestablishes operation
DT-based process
HRCA detail.
framework and advantages that each human/robot
advantages of
of human
human and
and robot
robot
to perform.
which
which reduce
reduce
A reasonable
assembly
assembly time
time
advantages
assignment of human
scheme and
can robot
give which
full play reduceto assembly
the time
respective
elaboratesCopyright
2405-8963 its design and operation
© 2022 The Authors. process
This isinandetail.
open access article under the CC BY-NC-ND license.
Peer review under responsibility of International Federation of Automatic Control. advantages of human and robot which reduce assembly time
10.1016/j.ifacol.2022.04.209
296 Yichen Wang et al. / IFAC PapersOnLine 55-2 (2022) 295–300

and cost (Chryssolouris et al., 2016). Xu et al. (2020) 3. DT-BASED HRCA DESIGN AND OPERATION
developed a mathematical model with disassembly time, cost FRAMEWORK
and difficulty as optimization objectives. The model was
Based on the current research, this paper constructs a DT-
described based on the precedence constraint relations of the
based HRCA design and operation framework, which
parts. Finally, a discrete seal algorithm was used to solve it.
consists of physical entities, virtual entities, twin data, service
Ranz et al. (2017) used an expert scoring method to evaluate
center, and component connections. These five components
H-R capabilities and assign tasks based on the results. The
are distributed in physical space and virtual space, which can
evaluation indicators include processing time, additional
provide design and operation services for HRCA process, as
investment and process quality.
shown in Fig.1.
The control of HRCA focuses on scheduling tasks and
controlling H-R execution with a combination of the plan and
the actual situation. Rüickert et al. (2018) used a kinect
camera to monitor the assembly process and display the task
and progress to the operator in real time. They also adjust the
robot's motion by detecting the range of human activities.
Fernández et al. (2017) integrated multiple sensors to sense
and control the assembly site. Humans can directly control
the robot through gestures, and the robot can adjust its speed
and posture in real time according to human behavior.
During the assembly process, it is also important to ensure
human and robot safety such as avoid collisions or other
accidents. Flacco Fabrizio et al. (2017, 2015) proposed a Figure 1. DT-based HRCA design and operation framework
method for dynamic environmental distance assessment
HRCA is mainly organized along operators (human and robot)
based on dual cameras which can accurately determine the
and operation objects (parts/assemblies), so they form the
distance between human and robot and take measures to
core of physical entities in the physical space. A variety of
avoid collisions in real time.
sensors and controllers are distributed around the core entities
The digital twin (DT) was first proposed by Professor to monitor and control the various states of the core entities
Michael Grieves at the University of Michigan (2020). The through real-time sensing. These sensors are the physical
conceptual model of DT consists mainly of a physical entity, medium for communication between the physical space and
a virtual entity and their connect. The control and the virtual space. In addition, there are also many other
optimization of the physical entity is achieved through real- entities that provide the necessary support for the HRCA.
time mapping, simulation and calculation in the virtual space.
The virtual entities in the virtual space are composed of
There is currently extensive research on the application of DT
multiple models, which are used to reflect the multi-
in manufacturing. Yi et al. (2021) presents a three layers
dimensional and multi-level states of the physical entities.
application framework for smart assembly process design
Among them, the geometric model and physical model reflect
based on DT. Liu et al. (2021) proposed a design architecture
the geometric shape, position, constraint relationship and
of flow-type smart manufacturing system based on DT. They
parameters of each physical entity from the spatial scale. The
also designed a DT system based on this structure.
behavior model and task model reflect the behavior and
There are also studies combining DT with HRCA. Arne action of each physical entity in the whole assembly process
Bilberg et al. (2019) carried out work based on DT on H-R from the time scale, which also reflect the assignment and
task assignment, real-time monitoring of the assembly execution of tasks.
process, and real-time robot control and trajectory
The twin data includes all the data generated and needed in
optimization. Lv et al. (2021) established a DT-based HRCA
the HRCA process. On the one hand, it is responsible for
framework from three aspects: overall elements, assembly
collecting and storing the data generated by multiple sensors
process and assembly performance. The experiments they
in the physical space for real-time updates of the virtual
performed suggest that the framework can improve assembly
entities. This data will also provide the data base for the
efficiency and safety.
services of the service center. On the other hand, the
The above studies addressed some main aspects of HRCA decisions made by the service center are downloaded to the
including layout design, task assignment and assembly twin database and translated into a stream of instructions to
control. However, there is lack of a unified framework for the the physical entities.
entire process from design to operation of HRCA based on
Based on twin data and virtual entity models, the service
DT. Meanwhile, existing research on DT places the focus on
center provides optimization and decision-making solutions
machines/robots and does not consider human
for HRCA design and operation by integrating multiple
comprehensively.
simulators and employing multiple algorithms. Before the
assembly starts, the service center first performs the design,
evaluation and optimization of the scenario; it also performs
 Yichen Wang et al. / IFAC PapersOnLine 55-2 (2022) 295–300
task assignment and assembly sequence planning based on
the respective capabilities of human and robot. During the
HRCA process, the service center mainly provides data
fusion, data visualization, event handling, task scheduling
and human-robot control to ensure safe and efficient
assembly.
The above four parts are interconnected through the
connection module, so that the digital twin model forms a
whole. The physical entity, virtual entity and twin data realize
real-time synchronization and virtual-real communication
through the connection. The twin data, virtual entities and
service center are connected to achieve rapid information
transfer and real-time processing. Real-time sensing, accurate
simulation and iterative optimization of the physical entity
state during the assembly process are realized through the
connection of physical entities and service centers.
4. KEY PROCESSES OF THE FRAMEWORK
4.1 DT-driven HRCA design
The layout of the sense and the sequence and assignment of
human/robot tasks need to be determined before assembly
(Hereinafter referred to as layout and assignment). A
reasonable layout can improve assembly efficiency, enhance
human comfort, increase the safety of HRCA, and reduce
production costs. Meanwhile, reasonable assignment can
shorten the assembly time, reduce the consumption caused by
human and robot waiting for each other, give full play to their
skills, and improve the quality of products. Therefore, the
services of HRCA design process are mainly divided into
scene layout design, evaluation and optimization services and
task assignment, assembly sequence planning services, as
shown in Fig. 2.
Figure 2. DT based HRCA layout design and task assignment
In the process of layout and assignment, if the layout is done
first without assignment, it is impossible to determine
indicators such as assembly time. Therefore evaluating the
rationality of the layout could not be done either. However, if
the assignment is done first without the layout, properties
297
such as accessibility of the robot cannot be determined, then
the assignment cannot be done either. Therefore, we design
the layout design-task assignment-layout evaluation-layout
optimization process to achieve alternate iterations and spiral
optimization of layout and assignment.
Layout design services are performed based on virtual
entities, which include geometric and physical models. The
designer builds scenes and performs layout design in the
virtual platform based on virtual entities. The main task of the
design is to determine the location of each entity and its
working area, such as: robot independent working area,
HRCA area, parts placement area, camera monitoring area,
etc. The design of location and area ensures that each entity
has sufficient space to function with each other and their
working areas do not interfere with each other.
The service of layout evaluation is based on a system of
indicators, as shown in Fig. 3. The evaluation is divided into
four main parts: efficiency, safety, cost, and human factors
engineering (HFE). Each part also has several sub-indicators.
The data corresponding to indicators can be obtained through
the simulation of the virtual platform, and some of them need
to be obtained after task assignment. Finally, the layout is
iteratively optimized by adjusting the location/number-
/working area of each entity according to the evaluation
results.
Figure 3. Scene layout evaluation indicator system
Task assignment services and sequence planning services are
performed after each layout design is completed. The main
process is divided into two phases: modeling and solving. In
the modeling phase, the robot is usually unable to perform the
complete assembly process of the part due to its end
limitation. Therefore, we break down the part assembly
process into positioning tasks and connection tasks which can
be performed completely by the human or robot. Then, task
precedence constraints are established based on task
breakdown and part precedence constraints. For each task we
also establish a table of attributes for human/robot execution,
including the time, feasibility, and complexity of the
respective human and robot execution.
Since a large number of parts leads to a huge number of
assembly sequences and task assignment schemes, the
assignment problem is a typical NP problem, which is usually
searched and solved using heuristic algorithms. Taking the
genetic algorithm as an example, in this paper, firstly, we
generate a number of initial sequences randomly. Next, each
sequence is evaluated to obtain its fitness. Subsequently,
selection, crossover and mutation operations are performed to
iteratively optimize until it converges. Finally, an optimized
solution is obtained. However, the algorithm does not fully
298 Yichen Wang et al. / IFAC PapersOnLine 55-2 (2022) 295–300
abstract the complete assembly process, which leads to
algorithmic results that are not always fully feasible, e.g., the
algorithm itself cannot consider the spatial conflicts when the
human and robot perform their respective tasks. Therefore,
we simulated the whole assembly process in the simulation
platform according to the optimization results and fine-tuned
the scheme to verify the feasibility and rationality of the
assignment.
The HRCA design based on DT avoids the repeated
adjustment of the actual layout and task assignment scheme
through full-true modeling of physical entities, multi-
algorithm optimization and full-flow simulation, which also
improves the reliability and rationality of the design.
4.2 DT-driven HRCA Operation
The HRCA process requires humans and robots to perform a
series of tasks in parallel or serially according to a task
assignment scheme. Besides, some other events (e.g., human-
robot collision, etc.) may occur during the assembly process
to affect the normal task execution or even threaten the
human-robot safety. Therefore, a control framework is
needed that can schedule tasks in real time according to their
execution and react quickly to different events.
In this paper, we propose a DT-based HRCA operation
framework, as shown in Fig. 4. The framework provides Data
Fusion and Visualization Service (DFVS), Event Handling
Service (EHS), Task Scheduling Service (TSS) and Human-
Robot Control Service (HRCS) in the operation process. We
view the HRCA process as a collection of tasks and events.
The service center monitors the different types of events
generated during the assembly process, which are then
processed in order of priority and finally issued to
humans/robots for execution as tasks.
Figure 4. DT-based HRCA operation framework
The Data Fusion and Visualization Service (DFVS) mines
and processes multivariate heterogeneous data and visualizes
the necessary data to facilitate human operations and
managerial supervision of the assembly process. During the
assembly process, the operator's voice, gestures,
electroencephalogram (EEG), and other information are
captured in real time by their respective sensors, which form
the sensing data and uploaded to the edge computing server
(ECS). ECS calculates these data using pre-set algorithms to
filter out the invalid information and upload the valid
information to the big data center while updating the virtual
entities. Finally, DFVS classifies and stores these data, and
makes certain inferences and predictions based on knowledge
graphs. At the same time, DFVS can visualize these data,
such as robot trajectory, task progress, operator status, etc. In
addition, the concurrent calculation method in ECS can
process a large amount of data simultaneously in a short
period of time, which achieves real-time information
synchronization between real and virtual space. What’s more,
the fast computation saves the arithmetic power of the central
control program and shortens the decision time.
The Event Handling Service (EHS) is responsible for
identifying and classifying the fused data. In this case, the
data is divided into three types: event data: commands,
warnings, collisions, etc.; status data: human fatigue status,
safety/hazard status, etc.; and prediction data: human
behavior, intentions, possible risks, etc. Subsequently, they
are divided into three types of events: planned events, regular
events, and abnormal events. In this paper, planned events are
events arranged according to the task assignment scheme.
During the assembly process, these events will be triggered
sequentially based on the task completion sequence. Planned
events make it easy for the control center to keep track of the
task progress and to schedule based on it. In 4.1 we
decomposed the part assembly process into positioning tasks
and connection tasks based on H-R capabilities. For fine-
grained control, this section further decomposes each task
into a series of continuous motions, as shown in Fig. 5. We
use the start and end of each motion as planned events.
Where the start event triggers the motion execution and the
end event triggers the start of the next motion. Both regular
and abnormal events are events detected by sensors that are
input by an external human or robot. Regular events are
usually events that are entered by the human/robot to assist
assembly, e.g., gestures, voice commands, etc., while
abnormal events are errors, failures, and dangerous events
that occur during assembly, e.g., robot disconnection, H-R
collision, or intrusion by an outside party, etc. In general, the
priority of the three events is: abnormal events > regular
events > planned events.
Task Scheduling Service (TSS) handles tasks based on task
priority. For dangerous abnormal events, TSS can create new
emergency tasks, e.g., robot emergency stop. For general
abnormal or regular events, TSS can pause/resume/change
tasks or extend/compress task execution time. For scheduled
events, TSS can start/end tasks according to task assignment
scheme.
 Yichen Wang et al. / IFAC PapersOnLine 55-2 (2022) 295–300
Figure 5. Positioning task and connection task breakdown
The Human/Robot Control Service (HRCS) performs the
corresponding kinematic/dynamic calculations based on the
tasks issued by the TSS, and finally translates these tasks into
specific commands issued to physical entities, which in turn
control the human/robot assembly. For the human, HRCS
control methods include output in the form of
video/picture/text to guide the operator deal with assembly.
For the robot, HRCS can change its trajectory, control end
grasping or releasing to match the human assembly. Through
the cooperation of the above four services, a series of events
in the HRCA process can be controlled and handled which
achieve tacit human-robot deep collaborative and ensure
efficient assembly.
5. CASE EXPERIMENT
A satellite assembly is shown in Fig. 6, which mainly
consists of a central straight cylinder, 14 plates, 16
connectors and several bolts. Due to the short assembly
period of satellite products, the assembly efficiency is very
demanding, but due to the many types of plates with high
similarity, the traditional manual assembly is prone to errors
which cause multiple rework. To overcome these drawbacks,
we implemented HRCA assembly of satellite panels
according to the DT-based HRCA design and operation
framework, as shown in Fig. 7.
Figure 6. Satellite plates assembly
During the pre-assembly design phase, the assembly layout
and tasks were designed and assigned in V-REP software,
and the results were optimized, as shown in the upper part of
Fig. 7. Subsequently, the physical space was laid out
completely according to the design and was calibrated using a
3D measurement arm.
According to the design, the physical entity mainly consists
of human, UR5 robot, parts, table, and screwdriver. The
monitoring devices mainly consists of: Kinect camera and
laser projector, where the camera is responsible for
identifying the plate type and position; the laser projector is
299
responsible for accurately projecting the plate position after
the plate assembly is completed to verify the assembly
accuracy. The control devices mainly consist of UR5 control
cabinet, infrared remote control, and monitor, they are
responsible for robot control, robot end control, real-time
display of assembly information and guiding human
assembly respectively.
Figure 7.An application scenario of collaborative H-R assembly
based on DT (a) Task assignment scheme; (b) Virtual reality
mapping during robot motion; (c) Monitor displaying information to
guide human assembly; (d) HRC assembly of the board; (e)
verification of assembly accuracy by laser projector;
We have added suction cups at the end of the robot, so the
robot is mainly responsible for picking up and positioning the
plates, while a human can perform both the plate positioning
and connection tasks. Now, we illustrate the operation of the
assembly with a board HRCA as an example, as follows.
Step1: The TSS issues assembly tasks according to the
predetermined assignment scheme.
Step2: The HRCS sends commands to control the robot
movement and pick-up the plate.
Step3: The robot grabs the plate and moves to the camera
position, where the camera identifies the plate type and
position and sends it back to the DFVS.
Step4: The virtual entity model is updated in real time. Then,
HRCS performs robot trajectory planning. The robot then
moves to the assembly position according to the planning
result. During the robot movement, the robot trajectory is
read by the monitoring device in real time and synchronized
in the virtual entity.
Step5: The monitor displays the task and guides the human to
fasten the plate. Then, the suction cup at the end of the robot
is released.
Step6: The robot is reset and the TSS detects the assembly
completion event and then releases the next task.
During the assembly process, we also consider taking some
measures to guarantee the safety of human and robot. In case
of a dangerous situation, human can press the emergency stop
300 Yichen Wang et al. / IFAC PapersOnLine 55-2 (2022) 295–300

button on the control panel to stop the UR5 movement
immediately; or press the infrared remote-control button to
manually control the UR5 end suction cup ventilation and
disconnection.
In the future, we can do more work based on this framework.
For example, we can add microphones so that human can
interact by voice. We can also use cameras to monitor
human-robot collisions; or let the robot decelerate its motion
when it enters the HRCA area, etc.
6. CONCLUSIONS
In this paper, we propose a framework for HRCA design and
operation based on digital twins, which consists of five
dimensions, i.e. physical entities, virtual entities, twin data,
service centers and their connections. Secondly, based on the
proposed framework, this paper details the design and
operation service mechanism of HRCA. Finally, we validate
the rationality of the framework by using satellite panel
assembly as an example. The results of the work show that
the framework was reasonable and implementable. Based on
the framework proposed in this paper, it avoids the repeated
adjustment of physical space in the layout and assembly
process on the one hand, and corrects the deviation between
the theoretical model and the actual assembly process on the
other hand, which reduces the assembly failure rate. Since
this paper did not complete all the functions of the proposed
framework, future work could be carried out based on it, e.g.,
VR/AR-based HRCA visualization guidance, multi-sensor
fusion for human/robot safety event detection and response,
etc.
ACKNOWLEDGEMENTS
This research is supported by Preliminary Research Program
of Equipment Development Department (61409230103).
REFERENCES
Bilberg, A., Malik, A.A., Digital twin driven human–robot
collaborative assembly. CIRP Annals, 68(1), 499-502,
2019.
Chryssolouris, G., Tsarouchi, P., and Makris, S. Human-
robot interaction review and challenges on task planning
and programming. International Journal of Computer
Integrated Manufacturing, 29(7/9), 916-931, 2016.
Fernández, J.G., Mronga, D. , Günther, M., Knobloch, T.,
and Kirchner, F.. Multimodal sensor-based whole-body
control for human–robot collaboration in industrial
settings. Robotics and Autonomous Systems, 94, 102-
119, 2017.
Flacco, F., and Luca, A. D.. Real-time computation of
distance to dynamic obstacles with multiple depth
sensors. IEEE robotics & automation letters, 2(1), 56-63,
2017.
Flacco, F., Kroeger, T., De Luca, A., and Khatib, O.. A depth
space approach for evaluating distance to objects.
Journal of Intelligent & Robotic Systems, 80(1), 7-22,
2015.
Hashemi-Petroodi, S. E., Thevenin, S., Kovalev, S., and
Dolgui, A. (2020). Operations management issues in
design and control of hybrid human-robot collaborative
manufacturing systems: a survey. Annual Reviews in
Control, 49, 264-276, 2020.
Liu, Q., Leng, J., Yan, D., Zhang, D., and Chen, X.. Digital
twin-based designing of the configuration, motion,
control, and optimization model of a flow-type smart
manufacturing system. Journal of Manufacturing
Systems, 58(B), 52-64, 2021.
Lv, Q.B., Zhang, R., Sun, X.M., Lu, Y.Q., and Bao, J.S. A
Digital Twin-Driven Human-Robot Collaborative
Assembly Approach in the Wake of COVID-19. Journal
of Manufacturing Systems, 60, 837-851, 2021.
Pooja, W., Varun, G., and Kerstin, J. Safe Layout Design
and Evaluation of a Human-Robot Collaborative
Application Cell through Risk Assessment – A
Computer Aided Approach. Procedia Manufacturing,
25, 602-611, 2018.
Ranz, F., Hummel, V., Sihn, W. Capability-based task
allocation in human-robot collaboration. Procedia
Manufacturing, 9, 182-189, 2017.
Rückert, P., Adam, J., Papenberg, B., Paulus, H., and Tracht,
K.. Calibration of a modular assembly system for
personalized and adaptive human robot collaboration.
Procedia CIRP, 199-204, 2018.
Tsubasa, K., Hamzeh, Reza., and Xu, X.. STEP-NC Enabled
Machine Tool Digital Twin. Procedia CIRP, 93, 1460-
1465, 2020
Wang, L., Gao, R., Váncza, J., Krüger, J., and Chryssolouris,
G. Symbiotic human-robot collaborative assembly.
CIRP Annals, 68(2), 701-726, 2019.
Xu, W., Tang, Q., Liu, J., Liu, Z., Zhou, Z., Pham, D.T.
Disassembly sequence planning using discrete Bees
algorithm for human-robot collaboration in
remanufacturing. Robotics and Computer-Integrated
Manufacturing, 62, 101860, 2020.
Yi, Y., Yan, Y., Liu, X., Ni, Z., and Liu, J.. Digital twin-
based smart assembly process design and application
framework for complex products and its case
study. Journal of Manufacturing Systems,58(B), 94-107,
2021.