StereoScan: Dense 3d in Real-time

Andreas Geiger, Julius Ziegler, Christoph Stiller

K ARLSRUHE I NSTITUTE OF T ECHNOLOGY

KIT – University of the State of Baden-Wuerttemberg and www.kit.edu

National Laboratory of the Helmholtz Association
 Contents

1 Motivation and Related Work

2 StereoScan: Approach

3 Experimental Evaluation / Future Work

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 2/24

 Contents

1 Motivation and Related Work

2 StereoScan: Approach

3 Experimental Evaluation / Future Work

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 3/24

 Motivation
stereo sequence 3d reconstruction

Goal:
Real-time 3d from stereo video on a single CPU
Applications:
Environment mapping / place recognition
Scene understanding

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 4/24

 Motivation
stereo sequence 3d reconstruction

Goal:
Real-time 3d from stereo video on a single CPU
Applications:
Environment mapping / place recognition
Scene understanding

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 4/24

 Related Work

[MonoSLAM, Davison et al.]

Simultaneous Localization and Mapping

Real-time systems exist
Mostly sparse features
Focus: loop-closure

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 5/24

 Related Work

[3d Recording for Archeological Fieldwork, Pollefeys et al.] [Building Rome in a Day, Agarwal et al.]

Structure-from-Motion
Monocular ⇒ requires motion
Multiple views of a single object
Computationally demanding

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 6/24

 Related Work

[Stixel World, Badino et al.]

Stixel World
Compact medium-level representation
Integration: Multiple Kalman filters
Can not represent overhanging structures
or strongly curved roads

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 7/24

 Contents

1 Motivation and Related Work

2 StereoScan: Approach

3 Experimental Evaluation / Future Work

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 8/24

 StereoScan - Overview

CPU with SSE3 instruction set

Core 1 12 fps Core 2 4 fps
Scene flow Dense stereo

Egomotion Fusion + 3d reprojection

frame 2
image plane

frame 1 frame 1

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 9/24

 StereoScan - Scene Flow

CPU with SSE3 instruction set

Core 1 12 fps Core 2 4 fps
Scene flow Dense stereo

Egomotion Fusion + 3d reprojection

frame 2
image plane

frame 1 frame 1

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 10/24

 StereoScan - Feature Detection

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 11/24

 StereoScan - Feature Detection

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 11/24

 StereoScan - Feature Detection

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 11/24

 StereoScan - Feature Detection

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 11/24

 StereoScan - Feature Description

16 locations within 11 × 11 block window

∂I ∂I
( ∂u , ∂v ) ⇒ 32 bytes per descriptor
Efficient Sum-of-Absolute-Differences (SAD) via SIMD

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 12/24

 StereoScan - Feature Matching
left image right image
e
tim

Feature matching:
Detect interest points using non-maximum-suppression
Match 4 images in a ’space-time’ circle
Use epipolar constraints for left ↔ right matching
Accept if last feature coincides with first feature

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 13/24

 StereoScan - Feature Matching
left image right image
e
tim

Feature matching:
Detect interest points using non-maximum-suppression
Match 4 images in a ’space-time’ circle
Use epipolar constraints for left ↔ right matching
Accept if last feature coincides with first feature

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 13/24

 StereoScan - Feature Matching
left image right image
e
tim

Feature matching:
Detect interest points using non-maximum-suppression
Match 4 images in a ’space-time’ circle
Use epipolar constraints for left ↔ right matching
Accept if last feature coincides with first feature

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 13/24

 StereoScan - Feature Matching
left image right image
e
tim

Feature matching:
Detect interest points using non-maximum-suppression
Match 4 images in a ’space-time’ circle
Use epipolar constraints for left ↔ right matching
Accept if last feature coincides with first feature

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 13/24

 StereoScan - Feature Matching
left image right image
e
tim

Feature matching:
Detect interest points using non-maximum-suppression
Match 4 images in a ’space-time’ circle
Use epipolar constraints for left ↔ right matching
Accept if last feature coincides with first feature

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 13/24

 StereoScan - Feature Matching

Fast feature matching:

1st: Match a sparse set of interest points within each class
Build statistics over likely displacements within each bin
Use this statistics for speeding up 2nd matching stage
Rejection outliers (Delaunay triangulation)

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 14/24

 StereoScan - Feature Matching

Fast feature matching:

1st: Match a sparse set of interest points within each class
Build statistics over likely displacements within each bin
Use this statistics for speeding up 2nd matching stage
Rejection outliers (Delaunay triangulation)

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 14/24

 StereoScan - Feature Matching

Fast feature matching:

1st: Match a sparse set of interest points within each class
Build statistics over likely displacements within each bin
Use this statistics for speeding up 2nd matching stage
Rejection outliers (Delaunay triangulation)

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 14/24

 StereoScan - Feature Matching

Fast feature matching:

1st: Match a sparse set of interest points within each class
Build statistics over likely displacements within each bin
Use this statistics for speeding up 2nd matching stage
Rejection outliers (Delaunay triangulation)

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 14/24

 StereoScan - Egomotion

CPU with SSE3 instruction set

Core 1 12 fps Core 2 4 fps
Scene flow Dense stereo

Egomotion Fusion + 3d reprojection

frame 2
image plane

frame 1 frame 1

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 15/24

 StereoScan - Egomotion

N 2 2
X (l) (r )
min xi − π (l) (Xi ; r, t) + xi − π (r ) (Xi ; r, t)

r,t
i=1

Minimize reprojection errors (Gauss-Newton + RANSAC)

Kalman Filter (constant acceleration model)

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 16/24

 StereoScan - 3D Fusion

CPU with SSE3 instruction set

Core 1 12 fps Core 2 4 fps
Scene flow Dense stereo

Egomotion Fusion + 3d reprojection

frame 2
image plane

frame 1 frame 1

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 17/24

 StereoScan - 3D Fusion

Greedy pixel association by reprojection into next frame

Depth fusion based on stereo uncertainty

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 18/24

 Contents

1 Motivation and Related Work

2 StereoScan: Approach

3 Experimental Evaluation / Future Work

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 19/24

 Experiments - Scene Flow

4
Kitt et al. 2010
3.5 our method
our method (2 scales) Stage Time
3
Filter 6.0 ms
Running time (s)

2.5
NMS 12 ms
2
Matching 1 2.8 ms
1.5 Matching 2 10.7 ms
1 Refinement 5.1 ms
0.5 Total time 36.6 ms
0
0 5000 10000 15000
Number of features

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 20/24

 Experiments - Scene Flow

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 21/24

 Experiments - Egomotion Estimation

1
10
Kitt et al. 2010
our method
0
10
Stage Time
Running time (s)

RANSAC 3.8 ms
−1
10
Refinement 0.4 ms
Kalman filter 0.1 ms
−2
10 Total time 4.3 ms

−3
10
0 50 100 150 200 250 300 350 400
Number of features

StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 22/24

 Experiments - Egomotion Estimation

80

70

60
z [meters]

50

40

30

20

10 GPS/IMU
Kitt et al. 2010
0
our method
−120 −100 −80 −60 −40 −20 0
x [meters]
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 23/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24
 Conclusion and Future Work
Conclusion:
Proposed a real-time 3D
reconstruction algorithm
Real-time on a single CPU
Large-scale stereo imagery
Code: www.cvlibs.net

Future Work:
Handle dynamic objects
Integrate multiple frames
3d urban scene
understanding (CVPR’11)
Thank you!
StereoScan: Dense 3d in Real-time – Source code: www.cvlibs.net 24/24