TRƯỜNG ĐẠI HỌC BÁCH KHOA HÀ NỘI

XỬ LÝ ẢNH TRONG CƠ ĐIỆN TỬ

Machine Vision

Giảng viên: TS. Nguyễn Thành Hùng

Đơn vị: Bộ môn Cơ điện tử, Viện Cơ khí

Hà Nội, 2021 1
 Chapter 9. Camera Calibration and 3D Reconstruction

1. Camera calibration

2. Robot Camera Calibration

3. Pose estimation

4. Stereo vision

2
 Chapter 9. Camera Calibration and 3D Reconstruction

1. Camera calibration

2. Robot Camera Calibration

3. Pose estimation

4. Stereo vision

3
 Camera calibration

❖ Geometric camera calibration, also referred to as camera resectioning, estimates the

parameters of a lens and image sensor of an image or video camera.
❖ You can use these parameters to correct for lens distortion, measure the size of an
object in world units, or determine the location of the camera in the scene.
❖ These tasks are used in applications such as machine vision to detect and measure
objects. They are also used in robotics, for navigation systems, and 3-D scene
reconstruction.

4
Camera calibration

5
 Camera calibration

❖ Camera parameters include intrinsics, extrinsics, and distortion coefficients.

❖ To estimate the camera parameters, you need to have 3-D world points and their
corresponding 2-D image points.
❖ You can get these correspondences using multiple images of a calibration pattern,
such as a checkerboard. Using the correspondences, you can solve for the camera
parameters.

6
 Camera calibration

❖ After you calibrate a camera, to evaluate the accuracy of the estimated parameters,
you can:
➢ Plot the relative locations of the camera and the calibration pattern
➢ Calculate the reprojection errors
➢ Calculate the parameter estimation errors.

7
 Camera calibration

❖ Pinhole Camera Model

➢ A pinhole camera is a simple camera without a lens and with a single small aperture.

8
 Camera calibration

❖ Pinhole Camera Model

9
 Camera calibration

❖ Pinhole Camera Model

▪ The calibration algorithm calculates the camera matrix using
➢ The intrinsic parameters
➢ The extrinsic parameters

Scale factor Image points Intrinsic Extrinsics World points

matrix Rotation and
translation
10
 Camera calibration

❖ Pinhole Camera Model

➢ The world points are transformed to camera coordinates using the extrinsics parameters. The
camera coordinates are mapped into the image plane using the intrinsics parameters.

11
 Camera calibration

❖ Camera Calibration Parameters

12
 Camera calibration

❖ Extrinsic Parameters
➢ The extrinsic parameters consist of a rotation, R, and a translation, t. The origin of the
camera’s coordinate system is at its optical center and its x- and y-axis define the image plane.

13
 Camera calibration

❖ Intrinsic Parameters
➢ The intrinsic parameters include the focal length, the optical center, also known as the
principal point, and the skew coefficient. The camera intrinsic matrix, K, is defined as:

➢ The pixel skew is defined as:

14
 Camera calibration

❖ Intrinsic Parameters

15
 Camera calibration

❖ Distortion in Camera Calibration

➢ The camera matrix does not account for lens distortion because an ideal pinhole camera does
not have a lens. To accurately represent a real camera, the camera model includes the radial
and tangential lens distortion.

16
 Camera calibration

❖ Radial Distortion
➢ Radial distortion occurs when light rays bend more near the edges of a lens than they do at its
optical center. The smaller the lens, the greater the distortion.

17
 Camera calibration

❖ Radial Distortion
➢ The radial distortion coefficients model this type of distortion. The distorted points are
denoted as (xdistorted, ydistorted):

•x, y — Undistorted pixel locations. x and y are in normalized image coordinates. Normalized
image coordinates are calculated from pixel coordinates by translating to the optical center and
dividing by the focal length in pixels. Thus, x and y are dimensionless.
•k1, k2, and k3 — Radial distortion coefficients of the lens.
•r2: x2 + y2

18
 Camera calibration

❖ Tangential Distortion
➢ Tangential distortion occurs when the lens and the image plane are not parallel. The
tangential distortion coefficients model this type of distortion.

19
 Camera calibration

❖ Tangential Distortion
➢ The distorted points are denoted as (xdistorted, ydistorted):

•x, y — Undistorted pixel locations. x and y are in normalized image coordinates. Normalized
image coordinates are calculated from pixel coordinates by translating to the optical center and
dividing by the focal length in pixels. Thus, x and y are dimensionless.
•p1 and p2 — Tangential distortion coefficients of the lens.
•r2: x2 + y2

20
 Camera calibration

❖ Camera model Camera coordinate system

Image plane

21
 Camera Calibration with OpenCV

❖ Function calibrateCamera use the above model to do the following:

➢ Project 3D points to the image plane given intrinsic and extrinsic parameters.
➢ Compute extrinsic parameters given intrinsic parameters, a few 3D points, and their
projections.
➢ Estimate intrinsic and extrinsic camera parameters from several views of a known calibration
pattern (every view is described by several 3D-2D point correspondences).
➢ Estimate the relative position and orientation of the stereo camera “heads” and compute the
rectification transformation that makes the camera optical axes parallel.

22
 Chapter 9. Camera Calibration and 3D Reconstruction

1. Camera calibration

2. Robot Camera Calibration

3. Pose estimation

4. Stereo vision

23
 Robot Camera Calibration
Eye To Hand Eye In Hand

Camera fixed to an independent structure Camera mounted on the robot

24
 Robot Camera Calibration
Eye To Hand Eye In Hand

A1 B1 A1
B1

B2
B2
A2 A2

Ai: transformation from Base → Gripper Ai: transformation from Base → Gripper
Bi: transformation from Target → Cam Bi: transformation from Cam → Target
X: transformation from Gripper → Target??? X: transformation from Gripper → Cam???
25
 Robot Camera Calibration
Eye To Hand

A1 B1 A1XB1 = A2XB2
A2-1A1XB1B1-1 = A2-1A2XB2B1-1
A2-1A1X = XB2B1-1
AX = XB
A = A2-1A1
B2
B = B2B1-1
A2

Ai: transformation from Base → Gripper

Bi: transformation from Target → Cam
X: transformation from Gripper → Target??? 26
 Robot Camera Calibration
Eye In Hand
A1 A1XB1 = A2XB2
B1
A2-1A1XB1B1-1 = A2-1A2XB2B1-1
A2-1A1X = XB2B1-1
AX = XB
A = A2-1A1
B2
B = B2B1-1
A2

Ai: transformation from Base → Gripper

Bi: transformation from Cam → Target
X: transformation from Gripper → Cam??? 27
 Robot Camera Calibration

➢ https://github.com/jhu-lcsr/handeye_calib_camodocal

➢ https://github.com/hengli/camodocal

➢ https://visp-doc.inria.fr/doxygen/visp-daily/classvpHandEyeCalibration.html#a68ab673312cc098bf8ed4f22b2d04d16

➢ https://github.com/zhixy/SolveAXXB

28
 Chapter 9. Camera Calibration and 3D Reconstruction

1. Camera calibration

2. Robot Camera Calibration

3. Pose estimation

4. Stereo vision

29
 Camera Pose Estimation

❖ 3D pose estimation is the problem of determining the transformation of an object in

a 2D image which gives the 3D object.
❖ 3D pose estimation from a calibrated 2D camera
➢ Given a 2D image of an object, and the camera that is calibrated with respect to a world
coordinate system, it is also possible to find the pose which gives the 3D object in its object
coordinate system.
➢ Starting with a 2D image, image points are extracted which correspond to corners in an
image. The projection rays from the image points are reconstructed from the 2D points so
that the 3D points, which must be incident with the reconstructed rays, can be determined.

30
 Camera Pose Estimation

❖ Algorithm
➢ The algorithm for determining pose estimation is based on the Iterative Closest Point
algorithm. The main idea is to determine the correspondences between 2D image features and
points on the 3D model curve.

(a) Reconstruct projection rays from the image points

(b) Estimate the nearest point of each projection ray to a point on the 3D contour
(c) Estimate the pose of the contour with the use of this correspondence set
(d) goto (b)

31
Camera Pose Estimation with OpenCV

32
 Chapter 9. Camera Calibration and 3D Reconstruction

1. Camera calibration

2. Robot Camera Calibration

3. Pose estimation

4. Stereo vision

33
 Stereo Vision

❖ What is stereo correspondence?

➢ only have 2D information when capture images
➢ the depth information is lost
➢ our brain takes two images and builds a 3D map using stereo vision
➢ capture two photos of the same scene using different viewpoints, and then match the
corresponding points to obtain the depth map of the scene → stereo vision algorithms

34
 Stereo Vision

❖ What is stereo correspondence?

Left Right
images images

35
 Stereo Vision

❖ What is stereo correspondence?

➢ The absolute difference between d1 and d2 is greater than the absolute difference between d3
and d4
➢ The camera moved by the same amount, there is a big difference between the apparent
distances between the initial and fnal positions → This happens because we can bring the
object closer to the camera; the apparent movement decreases when you capture two images
from different angles.
➢ This is the concept behind stereo correspondence: we capture two images and use this
knowledge to extract the depth information from a given scene.

36
 Stereo Vision

❖ What is epipolar geometry?

37
 Stereo Vision

❖ What is epipolar geometry?

Epipolar lines
Epipolar lines

38
 Stereo Vision

❖ What is epipolar geometry?

➢ Our goal is to match the keypoints in these two images to extract the scene information.
➢ The way we do this is by extracting a matrix that can associate the corresponding points
between two stereo images → the fundamental matrix.
➢ The point at which the epipolar lines converge is called epipole.
➢ If you match the keypoints using SIFT, and draw the lines towards the meeting point on the
left and right images, they will look like this:

39
 Stereo Vision

❖ What is epipolar geometry?

40
 Stereo Vision

❖ What is epipolar geometry?

➢ If two frames are positioned in 3D, then each epipolar line between the two frames must
intersect the corresponding feature in each frame and each of the camera origins.
➢ This can be used to estimate the pose of the cameras with respect to the 3D environment.
➢ We will use this information later on, to extract 3D information from the scene.

41
 Stereo Vision

❖ Building the 3D map

42
 Stereo Vision

❖ Building the 3D map

➢ The frst step is to extract the disparity map between the two images.
➢ Once we fnd the matching points between the two images, we can fnd the disparity by using
epipolar lines to impose epipolar constraints.

43
 Stereo Vision

❖ Building the 3D map

➢ x and x′ are the distance between points in image plane corresponding to the scene point 3D
and their camera center.
➢ B is the distance between two cameras (which we know) and f is the focal length of camera
(already known).
➢ So in short, above equation says that the depth of a point in a scene is inversely proportional
to the difference in distance of corresponding image points and their camera centers.
➢ So with this information, we can derive the depth of all pixels in an image.

44
 Stereo Vision

❖ Building the 3D map

45