Integration of

Vision and Inertial Sensing

PhD 2003 2007 InerVis
Integrating Inertial Sensors with Artificial Vision
Human inertial sensor: The vestibular system
key contributions:
a common framework for inertial-vision sensor integration;
calibration methods for integrated inertial and vision systems;
vertical feature segmentation and 3D mapping;
ground plane segmentation;
3D depth map registration;
independent motion segmentation. v t
x t
a t INS calcs
t t
body

g t world
Within the inner ear, the vestibular system measures tilt and angular acceleration

sensor data fusion

Data From Inertial Sensors
Inertial
sensors Match features
across images Dynamic 3D(t)
reconstructured
Camera world model

real world points

optical flow
Image
processing


MEMs Inertial sensors
lines Xsens MTi IMU
Images
time features time

Registering Stereo Depth Maps

Analog devices Analog devices ADXRS150 angular rate sensor

ADXL202 dual axis 2g accelerometer. (gyroscope). (October 2002)

Sensor Calibration
N static poses observing vertical target
Full camera
calibration

Independent Motion Segmentation IMUCAM Rotation estimated

Both sensors used to measure
Swinging pendulum sequence the vertical direction
N observations at different
camera positions
Background subtraction Unknown rotation determined

1)Quantise registered point cloud to voxel space and accumulate occupancy

votes for all frames 2N static poses with N rotations about IMU
2)Threshold to obtain background voxels (apply thinning and growing
transformation for noise filtering) IMUCAM Translation estimated
3)Intersect current frame voxels with complement of background voxels to
have voxels from moving objects - =
Raw voxels background voxels Seg. motion voxels

Optical flow consistency segmentation

1)Image optical flow (LK)
2)Estimate optical flow from Images
3D data and reconstructed
camera motion assuming static
scene
3)Subtract and threshold to
Camera
segment independent motion
motion
Depth
Map selected
selectedpublications:
publications:
Jorge
Jorge Lobo
Lobo andand Jorge
Jorge Dias,Dias, "Relative
"Relative Pose
Pose Calibration
Calibration Between
Between Visual
Visual and
and Inertial
Inertial
Sensors", International Journal of Robotics Research, Special InerVis Issue,
Sensors", International Journal of Robotics Research, Special InerVis Issue, in press. in press.
Peter
PeterCorke,
Corke,Jorge
JorgeLobo
Loboand andJorge
JorgeDias,
Dias,"An"Anintroduction
introductiontotoinertial
inertialand
andvisual
visualsensing",
sensing",
International Journal of Robotics Research, Special InerVis Issue,
International Journal of Robotics Research, Special InerVis Issue, in press. in press.
Luiz
LuizG.
G.B. B.Mirisola,
Mirisola,Jorge
JorgeLobo, Lobo,andandJorge
JorgeDias,
Dias,"Stereo
"StereoVision
Vision3D 3DMapMapRegistration
Registrationfor for
Airships using Vision-Inertial Sensing", In The 12th IASTED Int. Conf.
Airships using Vision-Inertial Sensing", In The 12th IASTED Int. Conf. on Robotics and on Robotics and
Applications,
Applications,Honolulu,
Honolulu,USA,
USA,August
August2006.
2006.
Jorge
Jorge Lobo,
Lobo, Joo
Joo Filipe
Filipe Ferreira
Ferreira and
and Jorge
Jorge Dias,
Dias, "Bioinspired
"Bioinspired Visuo-vestibular
Visuo-vestibularArtificial
Artificial
Perception System for Independent Motion Segmentation",
Perception System for Independent Motion Segmentation", In Second Inernational In Second Inernational
Cognitive
CognitiveVision
VisionWorkshop,
Workshop,ECCV ECCV9th 9thEuropean
EuropeanConference
Conferenceon onComputer
ComputerVision,
Vision,Graz,
Graz,
Austria, May 2006.
Austria, May 2006.
Jorge
Jorge Lobo
Lobo andand Jorge
Jorge Dias,Dias, "Relative
"Relative Pose
Pose Calibration
Calibration Between
Between Visual
Visual and
and Inertial
Inertial
Sensors", Proceedings of the ICRA 2005 Workshop on Integration of
Sensors", Proceedings of the ICRA 2005 Workshop on Integration of Vision and Inertial Vision and Inertial
Sensors
Sensors- -2nd
2ndInerVis,
InerVis,Barcelona,
Barcelona,Spain,
Spain,April
April18,
18,2005.
2005.
Jorge
JorgeLobo,
Lobo,Jorge
JorgeDias
Dias, ,Inertial
InertialSensed
SensedEgo-motion
Ego-motionfor for3D
3DVision,
Vision,ininJournal
JournalofofRobotic
Robotic
Systems Volume 21, Issue 1, pp. 3-12, January
Systems Volume 21, Issue 1, pp. 3-12, January 2004. 2004.
Jorge
JorgeLobo
LoboandandJorge
JorgeDias,
Dias,Vision
Visionand
andInertial
InertialSensor
SensorCooperation,
Cooperation,UsingUsingGravity
Gravityasasaa
Vertical
VerticalReference,
Reference,ininIEEE
IEEETransactions
Transactionson onPattern
PatternAnalysis
AnalysisandandMachine
MachineIntelligence,
Intelligence,
PAMI, 25(12), pp. 1597-1608, December
PAMI, 25(12), pp. 1597-1608, December 2003. 2003.
Jorge
JorgeLobo,
Lobo,Carlos
CarlosQueiroz,
Queiroz, Jorge
JorgeDias,
Dias,World
WorldFeature
FeatureDetection
Detectionand andMapping
Mappingusing
using
Stereovision and Inertial Sensors, in Robotics and Autonomous Systems,
Stereovision and Inertial Sensors, in Robotics and Autonomous Systems, Elsevier Elsevier
Science,
Science,vol.
vol.44,
44,Issue
Issue1,1,pp.pp.6981,
6981,July
July2003.
2003.

Contact Person:
Jorge Lobo
Email: jlobo@isr.uc.pt Mobile Robotics Laboratory
Institute of Systems and Robotics
ISR Coimbra