3D With Consumer Depth Sensors s1 Introduction 23T

3D with Consumer Depth Sensors
hands-on short course

Josep R. Casas
Image Processing Group (GPI)
Intelligent Data Science and Artificial Intelligence (IDEAI)
telecos BCN Spring Seminars

Jan 22nd – 26th 2024
3D with Consumer Depth Sensors Josep R. Casas
Short course goals
• Quick introduction to new concepts and techniques with commercial
consumer depth sensors: 3D capture, analysis and representation
• Hands-on!
Practical sessions include:
• data acquisition from the sensor
• real-time analysis, manipulation
• rendering and display
• tools from Point Cloud Library (PCL)
Thanks for their support with lab infrastructure go to:

Albert Gil (former GPI’s Software Eng.)
Joaquim Fernandez, Sergio Zarza (TSC IT & Technical Lab Support)
Francesc Guasch (ESETB IT Support)
3D with Consumer Depth Sensors telecosBCN - UPC Jan 22nd – 26th 2024 Josep R. Casas 2
Course contents
I. Introduction to 3D sensors
1.1 Standards & Applications
1.2 3D scene representation: images, depth, 3D sensors and multi-view
1.3 From 2D to 3D: multi-view, projection, reconstruction, scan
II. 3D sensors and range data

2.1 Types of 3D sensors
2.2 Range capture and depth data features: calibration
2.3 Range data representation: point clouds and meshes
Course contents (cont)
III. 3D data analysis
3.1 Filtering, registration, spurious removal
3.2 Estimation of simple geometric objects with RANSAC.
Comparison to geometry estimation from photometric data
IV. 3D reconstruction and scene creation

4.1 Rendering & display
4.2 Point cloud scenes, registration
V. Short course project

5.1 Solve a real-time challenge with a 3D sensor and the tools you have learnt
Course organization
K3D seminar @telecosBCN
Introduction to concepts and technologies related to depth sensors and data
Concepts Lab practice

22-Jan  Introduction  Getting started lab: e.g. code, point clouds
23-Jan  3D sensors and range data  Video matting with depth!
24-Jan  3D data analysis I  Geometry analysis (planes, spheres...)
25-Jan  3D data analysis II  Segmentation, normals, registration...
26-Jan  3D scene analysis/creation  Application project (assessment)
Check bibliography
C/C++ course
Initial References
[Machanick 2003] P. Machanick, C and C++ in Five Days, Univ. of Queensland 2011 (static link)
[Mutto 2012] C. dal Mutto, et al, Time-of-flight Cameras and Microsoft Kinect, Springer, 2012
[OpenNI] OpenNI, Natural Interaction Devices, structure.io/openni (formerly openni.org, now Apple!)
[Rusu 2011] R.B. Rusu, S. Cousins, 3D is here: Point Cloud Library (PCL), ICRA, 2011 pointclouds.org
[Zhang 2012] Z. Zhang, Microsoft Kinect Sensor and Its Effect, IEEE Multimedia 19(2), 2012
Technology evolution
Images
Video
Probably not this “3D”

…the key, here, is
1) technology, 3D???
and
2) availability to ALL
Why is this important?

2D imaging
• 2D imaging
“2D imaging projects 3D scenes onto a planar surface
(retina, sensor)… The depth (Z) dimension is lost.”
Camera obscura
M Pinhole camera
R
f
Fundamental ambiguity
9
• Any point P on the ray OP projects to the same point p in the image plane
Source: R. Collins
3D with Consumer Depth Sensors telecosBCN - UPC Jan 22nd – 26th 2024 Josep R. Casas
Camera Models
• The relation between an image point and its line of sight is given by the
camera model (e.g. pinhole camera) and the calibration parameters:
These parameters are called intrinsic camera parameters,
whereas the position & orientation of the camera are called extrinsic parameters
Pinhole camera model

X 
 x  f p x  1 
 R t   Y 
λ  y  =  f p y   1  0 T or λ x = P. X
1  Z 
 1   1   1   3  
1
Two eyes help!
• A second camera can resolve the ambiguity, enabling depth measurement via
triangulation
Camera projection parameters (intrinsic & extrinsic) are known
Source: R. Collins
Motivation and fundamental questions
Motto:
Computer Vision means “Teaching computers to see”
Antonio Torralba (MIT)
Talk@CVC 20th Anniversary, Barcelona July 9th, 2015
Exciting time for CV: new architectures, DBs, productivization, future
Fundamental problems: reconstruct 3D world, recognize…
Questions related to 2D/3D:
Q1: Is “projective vision” a natural way to capture the 3D world?
Q2: Do we need photometry to get geometry?
Q3: Does 3D vision mean the same than 3D geometry?
Q4: Does 2D/3D matter for “Teaching computers to see”?
On the rise…
Depth (range) sensors and technologies
Framos (website)
Time of flight measurement for range detection

© Analog Devices (website)
ARS 408-21 Long Range 77Ghz Radar

Tricam (website)
On the rise…
Point Cloud datasets for 3D processing
Range detection and people classification on time of flight measurement nuScenes (website)
Analog Devices (website)
Introduction to 3D Sensors and Range Data
3D vision comes from the concepts of:
• Projective transformations
perspective projection, projectivities (3D, plane-to-plane,
n-dimensional spaces, homography, invariants…)
• Multiple-View geometry
reconstruct real world scenes from several cameras (projections)
…or from a moving camera (SfM)
That’s a natural way into 3D vision as humans come equipped with:
• Two ‘projective’ sensors
• Multi-view (stereoscopic) vision
• And ‘Free Viewpoint View’ (FVV or rather “FV Acquisition”)
Projective vision ambiguities…
• Look twice
(c) 2018 jessicabackhaus.net (c) 2011 Santiago Bañón

…projectivity weakness used smartly
[Disclaimer: I’m not keen in bullfighting… at all!]
Pixabay (c) 2008 Fermin R. F. “Atento”
(c) 2010 Chema Conesa

(c) 2012elmundo.es
V.T.Altamirano
(c) 2018 jessicabackhaus.net
A1: Projective vision is natural? “yes” , practical? “Mmmm…”
2D imaging vs 3D imaging…
• 2D vs 3D
• Are there imaging sensors not projecting onto a plane?

• How can depth (range) be perceived directly?
• What would be the main advantage of capturing 3D directly?
…look into nature… active localization techniques…
Q2: Do we need to measure photometry to get geometry?

Introduction to 3D Sensors and Range Data
(cont)
• Distance measurement methods…
Range sensors (scanners!)
• 3D scanners and Lidar systems
• Light coding systems (e.g. Kinect™)
• TOF cameras (e.g. Kinect™ v2)
…and then
• Technology: pointwise scanners vs matricial imaging systems
• Will this result in a complete view… like a CAD design?
• Will we have Panoramic / Surround view? and Free viewpoint?
A2: Photometry to get geometry? Not necessarily…

Example of 3D capture
Lidar capture of the Stamford Bridge

FascinatE project, EU FP7-ICT 2010-2013
A3: Ups! “the same”? Bffff….
What can we do with 3D vision?
Example I of 3D processing (AR)
AR magic mirror using Kinect

T. Blum, N. Navab (2011) TUM
What can we do with 3D vision?
Example II of 3D processing (AR)
2015 Microsoft HoloLens - Transform your world with holograms

A4: Well, it may probably help… yes!
Answers to fundamental Questions
- Natural? Yes! Humans and most living beings are equipped with a pair of projective
(passive) sensors (eyes) performing stereoscopic vision to compute distances.
- Practical? Nope! Capturing 3D geometry can be better done with active sensors
probing the actual distance to scene surfaces.
Ex: far objects for automated driving with stereo vision!
Q2: Do we need to measure photometry to get geometry?

- Yes! For 2D projective imaging we do need photometry (depth from stereo or SfM)
to get scene geometry
- But No! Lidar or TOF sensors can compute scene geometry by radar principles, and
without resorting to disparity in photometric data.
And then…
- View is not complete!!!  single viewpoint occlusions…
- Panoramic/surround view is not FVV (usually viewpoint is fixed)
Answers to fundamental Questions (cont)
- Absolutely not!
- 3D vision usually assumes one (or several) points of view from which 3D geometry is
computed.
This leaves part of the scene geometry unavailable due to occlusions!
- Contrarily to Graphics or CAD design, where 3D geometry is a complete
representation of the scene (even rendering occlusions and transparencies)

- Geometry of objects and scenes in the 3D world can greatly help analysis and
recognition
- 3D geometry has the potential to avoid the large variability in appearance and
missing information due to projection, occlusions, motion, shadows...
- Yes, 3D geometry may definitely help computers to see and "understand" complex
visual scenes, reasoning about events that evolve in space and time
3D/range/scan sensors
Advantages 3D/range/scan Disadvantages
•… • …
…
…
• Actually not 3D but 2,5D!
• Direct measure of RGB+D
3D geometry Surface vs 3D (interface air-matter vs
volumetric 3D scanner, i.e. PET)
• Independent of photometry • Single viewpoint
(active sensors) Neither panoramic, nor surround view
Doesn’t hold for reflectivity  still need MultiView!!!
• Accuracy/resolution? Processing!
• From scan to CAD blueprints vs MPix cams!
Ideally! vs OpenCV!
Symmetric paradigm of 3D Data
Double nature of 3D data
Data
Geometric information 3D data
Pure data, measures.
Numerical representation of objects
 calculation, simulation, measurements
Rendering
… illumination + reflectivity
3D capture
Photometric information (images)
Can be rendered, displayed and presented visually,
by exploiting our perception capabilities
Visual  exploration, analysis and understanding Images
(same than looking at the physical world)
Example 1: Terrestrial Laser Scanning
Photo: Convento do Cristo
3D model: ArcHC 3D research group, Univ. Lisboa
Example 2: Digital Photogrammetry
Photo-mosaics
Photo Tourism: 2006 Savelly & Seitz UWA, Szeliski MS
Offline technique
(computationally intensive)
 Photosynth (MS)
 My3Dscanner
Photogrammetry
Photogrammetry refers to the practice of deriving 3D
measurements from photographs.
• Advances in digital cameras, processors, and computation,
yield extremely dense and accurate 3D surface data
from a limited number of photos with standard digital photography
• Structure from Motion (SfM) and sub-pixel image matching yield
3D-dimensional structure by analyzing projected 2D motion fields created by a
sequential change of position of the camera sensor relative to the object.
Photographic sequences are captured to maximize information available from the
change in viewpoint
• Resulting data sets are software platform-independent and can be reused
Disadvantage: computation time…
Photogrammetry examples
Example 3: new consumer depth sensors
SDK
YouTube
Gestoos use depth sensors raw data. Use Gestoos with your favorite sensor.
Robust hand tracking without initialization, no matter your body pose.
Static gestures with fast and robust detection. Create your own gesture.
Useful hand gestures, at various distances, without skeleton involved.
Commercial depth sensors: a revolution?
• Motivation
• Fact: Kinect had 10 million sales 3 months after launch Oct.2010
• New Guinness World Record:
’Fastest-Selling Consumer Electronics Device’
Potential for HCI

Potential for HCI
• Replacing well established paradigm of keyboard/mouse/screen
• Leveraging sensing technology to develop creative new ways to
interact with machines and to perform other tasks. . .
 from helping children with autism
 to assisting doctors in operating rooms
Depth sensors’ (Kinect’s) impact
• Understanding ’users’:
• when they talk
• who they are
• interpret their movements
• translate them into a format that developers can use to build ’new experiences’
• Applications in multiple industrial sectors
• entertainment
• education (edutainment?)
• healthcare
• retail
• transportation
Application examples
• Minority report/Oblong technologies
• Microsoft Project Natal
• CES2012 Samsung Smart TV
• FascinatE gesture control: dynamic & static
• Leap motion
More recently in AR/VR applications

• Microsoft Hololens
• Google Project Tango
• Occipital Bridge Engine
• Oculus Rift, HTC Vibe, etc.
Now…
• Is this the same 3D
as in Avatar?
• Could this be a technological revolution similar to digital images,

digital video… for digital 3D?
Cheap sensors + hi-res + processing tools!
What it does?
Instead of a 2D video like this one... ... it outputs pointclouds (“stream data”)
“cutting demo cloud”, Jeremie Papon PhD (c) 2014 k3d@k3d:~>> k3d-viewer-app Data/capture.oni
References (3D sensors)
[Bhandari 2016] A. Bhandari, R. Raskar, Signal processing for TOF imaging sensors: An
introduction to inverse problems in computational 3-D imaging, IEEE SPM 33(5), 2012
[Bishop 2012] E. Bishop, P. Favaro: The Light Field Camera: Extended Depth of Field,
Aliasing, and Superresolution. IEEE PAMI 34(5), 2012
[Foix 2011] S. Foix et al, Lock-in Time-of-Flight (ToF) Cameras: A Survey, IEEE Sensors
11(9), 2011
[McManamon 2012] P. McManamon, “Review of ladar: a historic, yet emerging, sensor
technology with rich phenomenology,” OE 51(6), 2012
[Salvi 2010] J. Salvi et al, A state of the art in structured light patterns for surface
profilometry, Pattern Recognition 43(8), 2010
[Smisek 2011] J. Smisek, et al, 3D with Kinect, ICCV Workshops 2011
[Xiong 2017] Z. Xiong, et al, Computational Depth Sensing : Toward high-performance
commodity depth cameras, IEEE SPM 34(3), 2017
See also [Mutto 2012] [Zhang 2012] from previous refs. page

3D With Consumer Depth Sensors s1 Introduction 23T

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3D With Consumer Depth Sensors s1 Introduction 23T

Uploaded by

Copyright:

Available Formats

3D with Consumer Depth Sensors

hands-on short course

telecos BCN Spring Seminars

Thanks for their support with lab infrastructure go to:

II. 3D sensors and range data

IV. 3D reconstruction and scene creation

V. Short course project

Concepts Lab practice

Probably not this “3D”

Why is this important?

Pinhole camera model

Time of flight measurement for range detection

ARS 408-21 Long Range 77Ghz Radar

(c) 2018 jessicabackhaus.net (c) 2011 Santiago Bañón

Pixabay (c) 2008 Fermin R. F. “Atento”

(c) 2010 Chema Conesa

• Are there imaging sensors not projecting onto a plane?

…look into nature… active localization techniques…

Q2: Do we need to measure photometry to get geometry?

A2: Photometry to get geometry? Not necessarily…

Lidar capture of the Stamford Bridge

AR magic mirror using Kinect

2015 Microsoft HoloLens - Transform your world with holograms

Q4: Does 2D/3D matter for “Teaching computers to see”?

Q2: Do we need to measure photometry to get geometry?

Q4: Does 2D/3D matter for “Teaching computers to see”?

Photo: Convento do Cristo

3D model: ArcHC 3D research group, Univ. Lisboa

Potential for HCI

More recently in AR/VR applications

• Could this be a technological revolution similar to digital images,

Cheap sensors + hi-res + processing tools!

You might also like