Proceedings of the 2002 IEEE

InternationalConference on Robotics 8 Automation

Washington, DC May 2002

Effects of Time Delay on Telerobotic Control

of Neutral Buoyancy Vehicles

J. Corde Lane, Craig R. Carignan, Brook R. Sullivan, David L. Akin, Teresa Hunt, and Rob Cohen

Space Systems Laboratory

University of Maryland
College Park, MD 20742

Abstract
The use of telerobots in remote locations allows humans
to extend their capabilities to distant and hazardous
environments. Although safer for the operator, this
extension of capabilities comes at a price. Due to
communication across great distances, experiencing time
delays of up to several seconds is common. Any time
delay in the humadrobot interaction can significantly
degrade the effectivenessof operator control. This paper
will looks at two common instances of human-in-the-loop
command of robots: command of vehicle motion and
command of a manipulator to perform an assembly task.
Results from three studies are summarized to show the
degradation in performance caused by varying levels of Figure 1: Secondary Camera and Maneuvering Platform,
time delay. SCAMP (Foreground) monitors while Ranger
1 Introduction (Background) performs an installation

For over 25 years the Space Systems Laboratory (SSL)
has used neutral buoyancy to simulate the microgravity
environment of space. By modifying the buoyancy of a
person or robot in water, apparent weightlessness can be
achieved. The SSL utilizes a 25-foot deep, %foot
diameter tank to accommodate representativespace
structures and robots. Many hours of space maintenance
tasks have been simulated, demonstratingassembly,
repair, and astronaut-robotic cooperation.
The Ranger telerobot, shown in Figure 1 was designed to
simulate on-orbit repair and maintenance tasks using four
manipulator arms. A grapple arm (not shown) connects
the robot with the worksite, providing mobility for the
vehicle to optimize its working position. The two
dexterous arms are then used to perform the required task.
A video arm (not shown) is used to provide the remote
operator with the desired view of the work area. Also
shown in Figure 1, the Secondary Camera and
Maneuvering Platform (SCAMP) can be flown to any
location to provide supplemental live video. Remote
operators commanded both vehicles over Ethernet via a
fiber optic tether.
0-7803-7272-71021$17.000 2002 IEEE
Research has shown how time delay in the control loop
between the operator and a robot can significantly
degrade operator performance [l] [2] [3]. Held [4] found
that even at a time delay of 0.3 seconds, operators would
decouple their commands from the robotic system s
response. Ferrell[5] showed that as time delay increased,
the completion time for a two-degree of freedom (DOF)
manipulator-positioningtask increased proportionally.
Black [6] showed similar results illustrating the increase
in task time with higher levels of delay for a 6DOF
manipulator task. Thompson [7] shows the relationship
of time delay and difficulty of the task to degrade
performance. Task difficulty was related to the degrees of
constraint when inserting an object. For example,
touching a flat plate has no degrees of constraint, i.e. there
are no physical restraints that would restrict motion.
Inserting a round peg into a long slot has two degrees of
constraint, preventing bolh translational and rotational
movement against the wall of the slot. A round peg-in-
hole has four constraints and a square peg has five, as the
only degree of freedom is parallel to the axis of the hole.
Thompson found that as the degrees of constraint
increase, a linear increase in completion time occurred;
2874

Authorized licensed use limited to: Oulu University. Downloaded on June 06,2022 at 13:50:28 UTC from IEEE Xplore. Restrictions apply.
this linear relationship was true at several values of time 2 Free Flight Control With Time Delay
delay. The effects of time delay were additive to the
increased constraint difficulty. Warrick [8] used Free-flight capability allows the robot to move to a
compensatory pursuit tracking task with small time delays number of locations, providing better monitoring views
for up to 0.32 seconds. Performance decreased as the and improved worksite access. This ability for a robot to
delay increased, even when the subjects were not aware of quickly and accurately move to a location is essential for
the delay. monitoring or interacting with a worksite.
Previous research has shown that at about one second of 2.1 SCAMP Tests with Simulated Delay
time delay, operators begin to switch their control strategy
from continually commanding and compensating for any The experiment conducted required two experienced
time delay to a move and wait strategy [ 5 ] . With a operators of SCAMP to fly the telerobot through a three-
several second time delay, operators will input a series of dimensional course. Figure 2 shows SCAMP and the four
commands, then wait out the time delay to observe the hula-hoops used to make the course. Under different
results before sending the next series of commands. This fixed time delays the operators were to fly SCAMP
move and wait strategy can significantly increase through the course in minimal time.
completion times of tasks.
In some cases, the time delay is not fixed, but fluctuates
with time. For example, most commands may be
received after one second. However, about every 30
seconds a single command may take 3 seconds to reach its
destination or may be dropped completely. This is
common when sending commands over the Intemet; a
variable time delay exists due to successive
communication packets being routed differently. It has
been found that a short variable time delay can be more Figure 2: The Test Course
detrimental than a longer fixed delay. One strategy held
commands for a fured time, which was longer than The course was made up of four 86 cm (34-inch) diameter
slowest time delay. A fixed, longer delay was traded for hoops suspended along the outside wall of the tank. The
the variable one. operator could take any path from one hoop to the next,
In this research, simulated time delay was modeled by however, SCAMP had to pass through the hoop in one
time stamping the commands made by the operator and direction. With SCAMP s 71 cm (28-inch) outer
saving them to a buffer. The simulation would then check diameter traversing through the hoops was not a simple
the time stamp periodically and determine if a sufficient task, Figure 3. The operators could maneuver SCAMP in
amount of delay had expired. If not, the command four degrees of freedom. SCAMP could translate in all
remained in the buffer until the time delay had passed. three directions: forwardhack, lewright, and upldown.
Then the simulation would execute the delayed command. Also the robot could turn in yaw to the lewright. The
This process continued, constantly adding and removing ability to pitch SCAMP up and down or perform a barrel
commands from the buffer. This method of generating roll was not available.
time delay was adequate for simulating a fixed time delay.
None of the quantitative studies in this work investigated
the effects of a variable time delay, which would require a
more featured algorithm.
There are numerous ways to ameliorate the effects of time
delay. Remote operators can use predictive displays that
can estimate where the robot will be when time delay has
expired [9] [ 101. Greater control can be given to the
robot, allowing it to independently perform a task with
either zero or minimal operator input [ 1 13 [3]. These
techniques have proven quite helpful, however, this paper
will focus on unmitigated time delay. The operator alone
is responsible for compensating for the time delay.

Authorized licensed use limited to: Oulu University. Downloaded on June 06,2022 at 13:50:28 UTC from IEEE Xplore. Restrictions apply.
To assist the operator, they were provided with three
video views, illustrated in Figure 4. Two fixed external
cameras provided overall situational awareness. Each
camera could view three of the four hoops. The third
view came from the onboard SCAMP camera.
(c)
Figure 4: Three views presented to the operator: (a)
onboard camera view, (b) (c) fixed camera views
To acquaint the operator with the task, they completed
two test runs at no time delay. The operators then
completed the course under each of the eight treatments
of time delay in the following order: 0,O.1,0.4,0.7, 1,
1.5,2,and 3 seconds. The operators performed two trials
at each treatment.
2876
Authorized licensed use limited to: Oulu University. Downloaded on June 06,2022 at 13:50:28 UTC from IEEE Xplore. Restrictions apply.
The analysis of variance (ANOVA) showed that time
delay had a significant effect on course completion time.
No significance or any interactions were found with
subjects or trials. Figure 5 shows the increase in
completion time due to time delay, averaged between
both operators and trials. The Duncan Range groupings,
at the 0.05 significant level are shown at the bottom of the
graph-
r
J
.g 300
0 0.5 1 1.5 2 2.5 3
GroupingAA A A A B B C
Time Delay [s]
Figure 5: Time Delay Effect on SCAMP Free Flight
No statistical significancewas found for one-second
delays and below. However at 1.5-second delay and
above a marked increase in delay occurred. This
corresponded with subjective comments, which reported
an increase in task difficulty as well as a transition to a
move, a wait strategy in controlling SCAMP. Although
completion times between 1.5 and 2 second delays were
not significantly different, 3-second delay completion
times were significantly higher.
Since the time delay increased with each successive test,
it is probable that learning may diminish the time delay
effect. This most likely occurred in the first (low time
delay) trials. The later tests had the benefit of learning
but the disadvantage of gTeater time delay. Retesting with
randomized time delay order may find a small increase in
completion times even at lower time delays.
2.2 SCAMP Operations Over the Internet
In four separate tests, SCAMP, located at Marshall Space
Flight Center WSFC), was alternately controlled from a
high school in Florida, the Johnson Space Center (JSC) in
Houston, and University of Maryland (UMD). In all cases
SCAMP was successful performing monitoring
operations,using remote control through the Internet,
even with the varying time delay. Figure 6 shows the
distribution of the remote control time delay. In all.cases,
the operators worked from satellite video that had about
250 ms delay.
0 0.25 0.5 0.75
Time Delay [SI
Figure 6 : Time Delay Distribution for Remote Control of
SCAMP
When controlling from the University (UMD -> MSFC),
the variation of time delay was small. About 97% of the
delay was less than 300 ms. This would have made
control easy, except the remaining 3% would range
between 1.5 - 6 seconds delay. To the operator, these
occasional long dropouts would appear as if the vehicle
had ceased functioning. Since they occurred about once
every two minutes, the operator could not anticipate the
dropout and would get frustrated.
Controlling SCAMP from Houston (JSC -> MSFC) the
variation was larger. The majority of delay was below
800 ms (98%). Although the typical time delay varied
more, there were fewer long duration dropouts than when
controlled from the University. Operator comments were
less concerned about these rare dropouts and instead
indicated a desire for less variability even at the expense
of average time deIay.
3 Robotic Manipulation With Time Delay
The ability for a robot to fly to a particular location allows
for many monitoring and docking tasks. However,
interacting with a worksite is essential in space
maintenance and assembly tasks. The goal of the Ranger
project is to demonstrate the ability of a robot to perform
tasks similar to those of a suited astronaut. This would
allow for a remote operator on the ground to work
cooperatively with an astronaut in space to perform
maintenance tasks. Clearly, communication time delay
between the ground operator and the space telerobot can
hamper this ability.
3.1 Manipulation Experimental Results
A study was performed using Ranger to perform a
changeout of a neutral buoyancy version of a space orbital
Authorized licensed use limited to: Oulu University. Downloaded on June 06,2022 at 13:50:28 UTC from IEEE Xplore. Restrictions apply.
replacement unit (ORU) fluids box, shown in Figure 7.
Two experienced operators performed multiple
changeouts under zero and three second time delay.
1
Figure 7: Ranger Performing an ORU Fluids Box
Changeout
An ORU fluids box changeout task is composed of
several steps. The trial would begin with Ranger s right
manipulator in a starting position near the installed ORU.
The operator would then control the manipulator, moving
the arm s tool tip to a H-Handle grasping fixture on the
ORU. After grasping the ORU, the operator would
activate the arm s tool drive which would in turn loosen
the bolt securing the ORU. Visual guides on the ORU
and its receptacle were provided to indicate when the
ORU was free. Finally the operator would move the arm
to extract the ORU from the worksite. After completing
the extraction task, the operator would then reinstall the
ORU by performing the above process in reverse: insert
the ORU into the worksite, tum the bolt to secure the
ORU, then release the ORU and move the arm away.
Several video cameras were available to assist the
operators to perform the task. Two fixed overall views
were used to monitor gross positioning of the arm. A
fixed close up view of the ORU receptacle allowed for
fine maneuvering of the manipulator and ORU. Finally,
SCAMP s free flying view would typically follow the
manipulator s tooltip as the task proceeded.
Each of the two operators performed about four complete
changeouts (extractions and insertions) at zero and three
second time delay. Figure 8 shows the increase in
completion time due to time delay. Using a generalized
linear model ANOVA, it was confirmed that time delay
did cause a significant increase in completion time, at the
0.01 level. It was found that the insertion task took
significantly longer, at a 0.05 level, than the extraction
task. This was due to the increase difficulty of inserting
the ORU into the worksite compared to the first step of
extraction by merely grasping the grapple fixture.
2877
 400
m
n
U

8
.3 300
, 3 sec T
w
c:
.3
z
&
200
E"
8 100 Figure 9: Overall view (Left) and right side close up view
(Right) from the Ranger Peg-In-Hole Simulation

0 The subjects tested under a range of time delays: 0, 0.5, 1,

Extraction Insertion
Time Delay [s]
Figure 8: Time Delay Effect on Telerobotic Extraction
and Insertion of a Fluids Box ORU
Subjective difference was also found significant, at a 0.01
level, as well as an interaction between subjects and task.
One subject had much higher completion times for the
insertion task, while the second subject extraction and
insertion tasks were much closer.
3.2 Manipulation Simulation Results
Several manipulator runs were also performed using a
graphical simulation [12]. One study used five subjects,
each testing for several hours in total. The task was to
insert the barebolt tool from a simulated Ranger
manipulator into a hole. Using the simulation as a more
controlled environment, the effect of time delay could be
emphasized. External variables like vehicle power,
changes in lighting, and friction variability in the arm s
motor could be eliminated.
The subjects operated the same input devices and
software used to control the actual Ranger manipulators.
However, they only controlled a simulation that was
displayed on the computer monitor. When the testing
started, the arm would move to an initial position. Due to
ease the programming constraints a simpler task was used
for the simulation. The subject would then command the
manipulator to drive the tip of the arm into a hole in the
box, show in Figure 9. When successfd, the simulation
would reset the ann to a different initial position and
another insertion would be attempted. A total of 32
insertions were performed on a given test. The subject
was able to toggle between three fvred views. An overall
view, shown in Figure 9, and two orthogonal close up
views of the hole for fine positioning. Only one of the
three views could be seen at a time.
1.5 and 3 seconds. So that learning would not become a
factor, each subject performed about 10 hours of training
on the simulator to become experienced with the task.
Then the subject would test each time delay treatment in
random order.
The timed results are displayed in Figure 10. Each
treatment of time delay was statistically significant from
each other, at the 0.01 level. This supported the work
performed by Held [4] and Warrick [SI, which indicated
that even small time delays could influence performance.
Also a linear trend with 0.983 correlation can be
established using the averaged data points. Since this task
isolated the importance of time delay, the increase in
delay greatly influenced completion times.
0 Ii
0 0.5 1 1.5 2 2.5 3
T h e Delay [SI
Figure 10: Linear Relationshipbetween Time Delay and
Completion Time
The completion time increased by 213% when going from
zero to 3 second time delay. This is markedly greater
than the ORU experimental data previously presented,
which had only a 47% increase. Part of this difference is
due to the elimination of environmental variables such as
lighting, views, etc. These variables cause perturbations
in the system making the task more challenging. Also
with only a few repetitions performed by each operator,
learning effects could have lowered the importance of
time delay in the ORU changeout task. Thus the task

2878

Authorized licensed use limited to: Oulu University. Downloaded on June 06,2022 at 13:50:28 UTC from IEEE Xplore. Restrictions apply.
became easier when performing the simulation,
emphasizing the effect of time delay.
Probably the most important factor was that the difference
between the two tasks made the simulation more
susceptible to time delay. Time delay affects positioning
tasks, and the simulation only used a positioning task.
The real ORU task not only required the arm to be
positioned, but also to grasp a fixture and turn a bolt.
These last two elements of the task are not affected by
time delay as much. Therefore the overall ORU
changeout task was less influenced by time delay.
4 Conclusions and Future Work
Studies presented here supported previous research that
time delay causes an increase in task completion times.
However, the level of degradation was dependent on the
task being performed. A three second delay caused a
132% increase in a free-flight maneuver task, only a 47%
increase in a manipulator maintenance task, and a 213%
increase using a simulated manipulator positioning task.
Compared to the free flight, where an increase for zero to
three-second time delay caused 132% increase in
completion time, the manipulative completion time
increase only 47%. Therefore the free flight operations
were influenced more by time delay than manipulator
operations. The greater effect on free flight was due to
the open loop control on SCAMP. With no input from the
operator, SCAMP would continue to drift. This was not
the case with Ranger s manipulator. The on-board closed
loop control would hold the manipulator to the last
commanded position. Therefore, when the operators
transitioned to a move and wait control strategy during
the free flight task, the vehicle would move out of
position making time delay compensation much more
difficult.
Because of the influence of several factors, characterizing
the importance of time delay during realistic applications
should be developed further. Although testing in
controlled environments and use of simulations is
important, the effects-of delay on remote control of
realistic and complex tasks can help guide how best to
ameliorate that delay. In addition, further research using
predictive displays can be used to help operators in both
manipulative [121 and free flight operations. The
combination of predictive displays with greater autonomy
on board the robot may create an effective combination to
alleviate the effects of time delay.
Acknowledgements
The authors would like to thank all the graduate and
undergraduate students who placed numerous hours on
these projects over the past 10 years. Their dedication
2879
Authorized licensed use limited to: Oulu University. Downloaded on June 06,2022 at 13:50:28 UTC from IEEE Xplore. Restrictions apply.
and perseverance has made the Space Systems Laboratory
a success over the years.
References
[ 11 Conklin, J. E. Effect of Control Lags on Performance
of a Tracking Task. Journal of Experimental
Psychology, 53(4) 1957.
[2] Sheridan, Thomas B. and William R. Ferrel. Remote
Manipdative Control With Transmission Delay. IEEE
Transactions Human Factors in Electronics,4. 1963.
[ 3 ] Sheridan, Thomas B. Telerobotics, Automation, And
Human Supervisory Control. Cambridge: MIT Press,
1992.
[4] Held R., A. Efstathiou, and M. Greene. Adaptation
to Displaced and DelayedVisual Feedback from the
Hand. Journal of Experimental Psychology, 72. 1966.
[5] Ferrell, William R. Remote Manipulation With
Transmission Delay. IEEE Transactions in Human
Factors in Electronics,6( 1). 1965.
[6] Black, J. H. Factorial Study Of Remote Manipulation
With Transmission Time Delay. M.S. Thesis MIT, 1971.
[7] Thompson, D. A. The Development Of A Six Degree
Of Freedom Robot Evaluation Test. Proceedings of 13th
Annual Conference Manual Control. MIT, Cambridge,
MA. 1977.
[8] Warrick, M. J. Effect of Transmission-Type Control
Lags on Tracking Accuracy. Wright-Patterson AFB,
OH: Aero Medical Laboratory. 1969. NASA Tech Report
5916.
[9] Hashimoto, T., Thomas B. Sheridan, and M.V. Noyes.
Effects of Predicted Information in Teleoperation
Through Time Delay. Japenese J.Ergonomics,22(2).
1986.
[IO] Noyes, Mark V. Superposition Of Graphics On Low
Bit Rate VideoAs An Aid In Teleoperation. M.S. Thesis
MIT, 1982.
[ 1 11 Backes, Paul G. Supervised Autonomy for Space
Telerobotics . American Institute of Aeronautics and
Astronautics, Inc., Washington, DC, 1994
[ 121 Lane, J. Corde. Human Factors Optimization of
VirtualEnvironment Attributesfor a Space Telerobotic
Control Station. Dissertation University of Maryland,
College Park. 2000.