!

!"#$%&'()(* ,%*-.$/*0(% 1(.

2"*(%(#("3 4%5-.6$*-. 7-80/9-3
!"#$% '"()*+, -."/)*+,01)2"+34".56
:"*90%-
! Mouvauon
! 1he underwaLer uomaln
! Plgh lrequency Sonar lmage rocesslng
! underwaLer 8obouc SlmulaLor
! Pardware lmplemenLauon
!"#$% '"()*+, -."/)*+,01)2"+34".56
;(*0<$*0(%
! Puman dlver Lasks:
! Collecung samples on coral
reefs
! Collecung samples from
dangerous [ellysh
! underwaLer cable
malnLenance
! lnspecuon of shlp hulls
! Weldlng underwaLer
sLrucLures
! underwaLer mlne dlsposal
! Salvage
!"#$% '"()*+, -."/)*+,01)2"+34".56
4%5-.6$*-. :=-.$*0(%$9 >8$99-%?-3
! Acousuc Communlcauons
! Low daLa raLe / narrow
bandwldLh
! MulupaLh propagauon
! Locallzauon
! CS denled envlronmenL
! vlslblllLy
! LlmlLed ln mosL of Lhe ocean
due Lo LurbldlLy
! Ccean waves and currenLs
! 8lofoullng
! lnhlblLs long endurance
mlsslons
!"#$% '"()*+, -."/)*+,01)2"+34".56
'-9$*-5 @(.A
! A vlslon-based ConLrol and
lnLeracuon lramework for a
Legged underwaLer 8oboL"
! 8oboLChaL C8 code
language
! erformed human
experlmenLs Lo measure Lhe
exLra work requlred by Lhe
dlver
A Vision-based Control and Interaction Framework
for a Legged Underwater Robot
Junaed Sattar and Gregory Dudek
Center for Intelligent Machines, McGill University,
3480 University Street, Montr eal, H3A 2A7
{junaed,dudek}@cim.mcgill.ca
Abstract
We present a vision-based control and interaction frame-
work for mobile robots, and describe its implementation in
a legged amphibious robot. The control scheme enables the
robot to navigate, follow targets of interest, and interact
with human operators. The visual framework presented in
this paper enables deployment of the vehicle in underwater
environments along with a human scuba diver as the op-
erator, without requiring any external tethered control. We
present the current implementation of this framework in our
particular family of underwater robots, with a focus on the
underlying software and hardware infrastructure. We look
at the practical issues pertaining to system implementation
as it applies to our framework, from choice of operating sys-
tems to communication bus design. While our system has
been effectively used in both open-ocean and closed-water
environments, we perform some quantitative measurements
with an effort to analyze the responsiveness and robustness
of the complete architecture.
1 Introduction
A human-robot interaction mechanism is an essential
component for large-scale adaptation of robots in everyday
life. For robots to act as assistants to humans, from house-
hold applications to industrial deployments, an uncompli-
cated, easy to use and intuitive interaction scheme is of
utmost importance. In this particular work, we look at a
vision-based interaction and control scheme, and describe
the implementation of this framework in the Aqua family
of underwater robots [5]. We choose vision as an interac-
tion modality since it creates opportunities for having many
of the sought-after qualities we mention above, as well as
being unobtrusive, energy-efcient and widely deployable
in many different domains. In our particular robot, the un-
derwater domain renders the traditional radio and wireless
communication methods useless, since the transmission en-
ergy is absorbed rapidly by the water medium, in particular
by sea water. Tactile sensing is possible, but that requires
extremely close proximity to the human operator and dis-
turbs the dynamics of the vehicle. Vision-based interaction
avoids all such issues, and with the aid of ducials makes
it possible to robustly and accurately detect symbols (and
accordingly instructions) coming from the human operator.
Our communication scheme has minimal deployment over-
head, which, in case of scuba divers (Fig. 1), is particu-
larly important since the diver would be under substantive
cognitive load already; for example maintaining the life-
supporting dive gear. The vision-based control and navi-
gation mechanisms enable the robot to follow targets using
a variety of approaches for example a scuba diver using
frequency-domain tracking.
Figure 1. The Aqua robot operating tether-
lessly on visual stimuli: a scuba diver start-
ing the robot by showing a tag.
For robust communication between a robot and its hu-
man operator, interactions are necessary at different tempo-
ral scales. That is, communication must exist in a broad
spectrum of invocation rates, ranging from frequently-
!"#$% '"()*+, -."/)*+,01)2"+34".56
'-9$*-5 @(.A
! underwaLer Puman-8oboL
lnLeracuon vla 8lologlcal
Mouon ldenucauon"
! use of lourler analysls Lo
deLecL and Lrack dlver.
! unscenLed kalman lllLer for
non-llnear esumauon of
dlver Lra[ecLory
Underwater Human-Robot Interaction via
Biological Motion Identication
Junaed Sattar and Gregory Dudek
Center for Intelligent Machines
McGill University
Montr eal, Qu ebec, Canada H3A 2A7.
Email: {junaed, dudek}@cim.mcgill.ca
AbstractWe present an algorithm for underwater robots to
visually detect and track human motion. Our objective is to
enable human-robot interaction by allowing a robot to follow
behind a human moving in (up to) six degrees of freedom.
In particular, we have developed a system to allow a robot
to detect, track and follow a scuba diver by using frequency-
domain detection of biological motion patterns. The motion of
biological entities is characterized by combinations of periodic
motions which are inherently distinctive. This is especially true
of human swimmers. By using the frequency-space response of
spatial signals over a number of video frames, we attempt to
identify signatures pertaining to biological motion. This technique
is applied to track scuba divers in underwater domains, typically
with the robot swimming behind the diver. The algorithm is able
to detect a range of motions, which includes motion directly away
from or towards the camera. The motion of the diver relative
to the vehicle is then tracked using an Unscented Kalman Filter
(UKF), an approach for non-linear estimation. The efciency of
our approach makes it attractive for real-time applications on-
board our underwater vehicle, and in future applications we
intend to track scuba divers in real-time with the robot. The
paper presents an algorithmic overview of our approach, together
with experimental evaluation based on underwater video footage.
Fig. 1. An underwater robot servoing off a colored target carried by a diver.
I. INTRODUCTION
Motion cues have been shown to be powerful indicators
of human activity and have been used in the identication
of their position, behavior and identity. In this paper we
exploit motion signatures to facilitate visual servoing, as part
of a larger human-robot interaction framework. From the
perspective of visual control of an autonomous robot, the
ability to distinguish between mobile and static objects in
a scene is vital for safe and successful navigation. For the
vision-based tracking of human targets, motion patterns are
an important signature, since they can provide a distinctive
cue to disambiguate between people and other non-biological
objects, including moving objects, in the scene. We look at
both of these features in the current work.
Our work exploits motion-based tracking as one input cue to
facilitate human-robot interaction. While the entire framework
is outside the scope of this paper, an important sub-task for
our robot, like many others, is for it to follow a human
operator (as can be seen in Fig.1). We facilitate the detection
and tracking of the human operator using the spatio-temporal
signature of human motion. In practice, this detection and
servo-control behavior is just one of a suite of vision-based
interaction mechanisms. In the context of servo-control, we
need to detect a human, estimate his image coordinates (and
possible image velocity), and exploit this in a control loop.
We use the periodicity inherently present in biological motion,
and swimming in particular, to detect human scuba divers.
Divers normally swim with a distinctive kicking gait which,
like walking, is periodic, but also somewhat individuated. In
many practical situations, the preferred applications of UAV
technologies call for close interactions with humans. The
underwater environment poses new challenges and pitfalls
that invalidate assumptions required for many established
algorithms in autonomous mobile robotics. While truly au-
tonomous underwater navigation remains an important goal,
having the ability to guide an underwater robot using sensory
inputs also has important benets; for example, to train the
robot to perform a repetitive observation or inspection task, it
might very well be convenient for a scuba diver to perform
the task as the robot follows and learns the trajectory. For
future missions, the robot can use the information collected by
following the diver to carry out the inspection. This approach
also has the added advantage of not requiring a second person
teleoperating the robot, which simplies the operational loop
and reduces the associated overhead of robot deployment.
Our approach to track scuba divers in underwater video
footage and real-time streaming video arises thus from the
!"#$% '"()*+, -."/)*+,01)2"+34".56
BC D(.6$.5&E((A0%? F(%$. ,#$?-.G
!"#$% '"()*+, -."/)*+,01)2"+34".56
H$30/ >(#="*-. 7030(% 29?(.0*8#3
! SLandard CpenCv AlgorlLhms 1esLed
! 1hresholdlng
! 1emplaung
! Cpucal llow
! CpenCv AlgorlLhms LhaL falled
! Cood leaLure Lo 1rack
! Pough Clrcles
! Lroslon / ullauon

!"#$% '"()*+, -."/)*+,01)2"+34".56
I8.-38(950%?
!"#$% '"()*+, -."/)*+,01)2"+34".56
! 1hresholdlng
! ConverL 8C8 lmage Lo gray scale
! 1hreshold Lo zero
I-#=9$*0%?
!"#$% '"()*+, -."/)*+,01)2"+34".56
! 1emplaung
! normed cross correlauon
! no Lracklng lmplemenLed yeL
:=*0/$9 D9(6
!"#$% '"()*+, -."/)*+,01)2"+34".56
! Cpucal llow
! 1hresholdlng performed rsL
! MovemenL of sonar resulLs ln
false posluves
J-.0(50/ ;(*0(% C-*-/*0(%
!"#$% '"()*+, -."/)*+,01)2"+34".56
Equations for Human Labs Presentation
November 12, 2012
Kevin DeMarco <demarco@gatech.edu>
Equations for Human Labs Presentation
Paragraph here.
S
t
1
,t
2
=
X
(x,y)B
t
1
|O
t
1
(x, y) O
t
2
(x, y)|
1
8. CuLler. 8obusL 8eal-1lme erlodlc Mouon ueLecuon, Analysls, and Appllcauons"

lrame slmllarlLy calculauon
KL$#=9- J-.0(50/ ;(*0(% C-*-/*0(%
!"#$% '"()*+, -."/)*+,01)2"+34".56
lrame number
l
r
a
m
e

n
u
m
b
e
r

8ecurrence loLs
J-.0(50/ C-*-/*0(% (1 F60##-.
!"#$% '"()*+, -."/)*+,01)2"+34".56
! 8ecurrence loL
! 1.) lace cenLrold of ob[ecL ln Lhe cenLer of
frame sequence
! 2.) use Lhe slmllarlLy equauon Lo compuLe
Lhe recurrence ploL
CenLered Sequence
8ecurrence loL
Aer Lhreshold
8ecurrence loL
H$/A?.("%5 F")*.$/*0(%
! Separauon of foreground"
wlLh background"
! Many algorlLhms rely on
sLauc vlew
Equations for Human Labs Presentation
November 14, 2012
Kevin DeMarco <demarco@gatech.edu>
Equations for Human Labs Presentation
Similarity Calculation
S
t
1
,t
2
=
X
(x,y)B
t
1
|O
t
1
(x, y) O
t
2
(x, y)|
Running Gaussian Background Subtraction

t
= I
t
+ (1 )
t1

2
t
= (I
t

t1
)
2
+ (1 )
2
t1
Threshold test:
|I
t

t
| > k
t
|I
t

t
| > k
t
+
1
!"#$% '"()*+, -."/)*+,01)2"+34".56
1hreshold 1esL:
1hreshold 1esL w/ oseL:
8unnlng Causslan
8ackground SubLracuon
'"%%0%? M$"330$% H$/A?.("%5
F")*.$/*0(%
!"#$% '"()*+, -."/)*+,01)2"+34".56
I6( J$33 H9() C-*-/*0(%
! 8lob ueLecuon kernel
!"#$% '"()*+, -."/)*+,01)2"+34".56
Equations for Human Labs Presentation
November 14, 2012
Kevin DeMarco <demarco@gatech.edu>
Equations for Human Labs Presentation
Similarity Calculation
S
t1,t2
=
X
(x,y)Bt
1
|O
t1
(x, y) O
t2
(x, y)|
Running Gaussian Background Subtraction

t
= I
t
+ (1 )
t1

2
t
= (I
t

t1
)
2
+ (1 )
2
t1
Threshold test:
|I
t

t
| > k
t
|I
t

t
| > k
t
+
Two Pass Blob Detection
Algorithm 1 Two Pass Blob Detection
1: procedure BlobDetect(image)
2: label 1
3: labelTable empty
4: for i 0, rows do
5: for j 0, columns do
6: if pixel is in foreground then
7: min ndMin(NE,N,NW,W)
8: if min = 0 then
9: pixel label
10: labelTable append label
11: label label + 1
12: else
13: pixel min
14: LabelNeighbors(min)
15: end if
16: end if
17: end for
18: end for
19: for i 0, rows do
20: for j 0, columns do
21: ResolveBlobMatch(labelTable,pixel)
22: end for
23: end for
24: end procedure
1
H9() C-*-/*0(% '-3"9*3
!"#$% '"()*+, -."/)*+,01)2"+34".56
1
2
3
4
3
6
7
8
9
F(%$. J.(/-330%? C$*$ D9(6
!"#$% '"()*+, -."/)*+,01)2"+34".56
F(%$. H9() C-*-/*0(%
!"#$% '"()*+, -."/)*+,01)2"+34".56
;:'FK F0#"9$*(.
! A generlc open source slmulaLor for
academlc roboucs
! 8ased on Lhe 8lender Came Lnglne
! SupporLs exLernal mlddleware
! 8CS, MCCS, ?A8
! Laslly develop new sensors and
robouc plauorms
!"#$% '"()*+, -."/)*+,01)2"+34".56
!"#$% '"()*+, -."/)*+,01)2"+34".56
F(%$. ;(5-9
!"#$% '"()*+, -."/)*+,01)2"+34".56
!"#$% '"()*+, -."/)*+,01)2"+34".56
F0#"9$*-5 F(%$. C$*$
!"#$% '"()*+, -."/)*+,01)2"+34".56
'-#(*-9G :=-.$*-5 7-80/9- I-3*0%?
! vldeo8ay ro 4
! 1eLhered Lo surface
! Plgh denluon camera for
ground LruLh"
! ueveloplng boLh a MCCS
appllcauon and 8CS node for low
level conLrol and sensor monlLorlng
! need Lo develop a conLrol law LhaL
maxlmlzes pose deLecuon of dlver.
!"#$% '"()*+, -."/)*+,01)2"+34".56
2"*(%(#("3 4%5-.6$*-. 7-80/9- I-3*0%?
!"#$% '"()*+, -."/)*+,01)2"+34".56
! C18l ?ellown Auv
! ueveloped nose cone Lo house 8luevlew 900
! Poverlng wlll be dlmculL due Lo underacLuaLed
sysLem.
;0330(% :.0-%*-5 :=-.$*0%? F"0*- N;::FO
! Cpen Source publlsh and
subscrlbe mlddleware
sysLem slmllar Lo Lhe 8oboL
Cperaung SysLem (8CS)
! plvPelm behavlor fuslon
module
! ueslgned for mlsslon
lmplemenLauon
!"#$% '"()*+, -."/)*+,01)2"+34".56
;0330(% :.0-%*-5 :=-.$*0%? F"0*- N;::FO
!"#$% '"()*+, -."/)*+,01)2"+34".56
D".*8-. @(.A
! lurLher lnvesugaLe background subLracuon
! lurLher lnvesugaLe frequency analysls of sonar lmagery
! lnvesugaLe movlng camera background subLracuon
! lmplemenL non-llnear dlver Lracker
! unscenLed kalman lllLer
! arucle lllLer
! Compare slmulaLed sonar daLa Lo acLual sonar daLa
! lmplemenL low-level conLroller on vldeo8ay Lo maxlmlze
condence of dlver pose deLecuon
!"#$% '"()*+, -."/)*+,01)2"+34".56
P"-3*0(%3Q
!"#$% '"()*+, -."/)*+,01)2"+34".56