2020 17th International Conference on Ubiquitous Robots (UR)

June 22-26, 2020. Kyoto, Japan

Image Projection onto Flat LiDAR Point Cloud Surfaces

to Create Dense and Smooth 3D Color Maps
Sumin Hu1 , Seungwon Song1 , Hyun Myung1∗

Abstract— This paper proposes an area-wise method to build

aesthetically pleasing RGB-D data by projecting camera images
onto LiDAR point clouds corrected by Graph SLAM. In
particular, the focus is on projecting images to corresponding
flat surfaces, extracted as plane equations by RANSAC. The
newly created data boasts a camera-like view even in 3D due
to its dense, yet smooth flat point clouds. However, since this
method is only limited to planar surfaces, other 3D data points
that could not be separated as planes had to suffer poor quality
due to sparse and rough LiDAR point clouds.

I. INTRODUCTION
In recent years, more advanced mobile robots, from
robotic vacuum cleaners to autonomous cars, are making
their debut in the commercial market. These robots heavily
depend on sensors such as LiDARs and cameras to map their
environments for navigation.
However, the problem with the current mapping mecha-
nisms is that the map created by a robot is primarily limited
to its original usage: Navigation for the robot. In other
words, that map of the environment cannot be used for other
purposes. Instead, if cars, for example, were to collectively
gather 3D maps of the environment, a complete and current
street view would be always accessible for other uses.
That limitation in secondary usage of the 3D maps greatly
undermines the robots’ versatility. Accurate, automatic, and Fig. 1: Turtlebot2 with an Intel NUC, a Velodyne Puck, and
aesthetic 3D map generation would make tedious efforts a monocular camera
like taking pictures every few meters to create a street view
obsolete and furthermore with machine learning, the raw data
would provide many folds more than what we can currently
offer with 2D maps.
Additionally, 3D maps need to be aesthetically pleasing if With SLAM’s versatile applicability and variations, there
we were to bring 3D maps as a service to end users. How- has been numerous research from improving SLAM algo-
ever, most 3D data points acquired are rarely processed for rithms in unfavorable situations [2], [3], [4] to utilizing
aesthetics since it’s time consuming and doing so becomes distinctive sensors [5] and their combinations [12], to name
a whole new project on top of the original. a few. Additionally, some narrow their application to charac-
Simultaneous Localization And Mapping (SLAM) [1] , as teristic environments, whether it be indoors [6] or outdoors;
the name suggests, is a method for mapping the surrounding, aerial [7] , aquatic [8], or terrestrial. This paper, in particular,
synchronous to locating itself in it. It has been the center focuses it’s application to indoor environments, which are
of robot mobility research for the past two decades. Its heavily surrounded with numerous plane surfaces called
value comes from the fact that it does not rely on a global walls.
infrastructure level of technology requirement such as the
GPS, but does so with just a few sensors on the robot to Due to the numerous walls an indoor environment inher-
work. Yet, the system can map and navigate the environment ently has, many research [9], [10], [11] exploit the features
with greater accuracy than utilizing GPS. of flat planes to their advantage. Similarly, observing how
a projector projects movies onto a flat screen, we con-
1 Sumin Hu, Seungwon Song, and Hyun Myung are with the Ur-
ceptualized a straightforward 3D map generation method
ban Robotics Lab, Korea Advanced Institute of Science and Technology
(KAIST) Daejeon, 34141, South Korea (e-mail: 2minus1@kaist.ac.kr, ss- that projects images onto extracted flat planes, therefore
wan55@kaist.ac.kr, hmyung@kaist.ac.kr). manifesting richer and more human friendly visual maps.

978-1-7281-5715-3/20/$31.00 ©2020 IEEE 459

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on October 15,2021 at 00:38:56 UTC from IEEE Xplore. Restrictions apply.
 II. PROCEDURE
A. Sensor Fusion (from Sensors to Robot to Global)
1) Camera to Robot
pR = MCR pC
where pR and pC are the positions of the robot and
the camera, respectively, and MCR is the homogeneous
transformation matrix which describes the frame of the
camera relative to that of the robot.
2) 3D LiDAR to Robot
pR = MLR pL
where pL is the position of the 3D LiDAR and MLR
is the homogeneous transformation matrix which de-
scribes the frame of the 3D LiDAR relative to that of
the robot.
3) Robot to Global
pG = MRG pR
where pG is the position of the global frame and
MRG is the homogeneous transformation matrix which
describes the robot frame relative to the global frame.
B. Applying GraphSLAM
GraphSLAM [13] and ICP (iterative closest point) algo-
rithm were used to make the map since we are using a 3D
LiDAR in this paper. Every unit distance the robot moves, a
node is created and the relative distance between the previous
node and the current node acts as a weak constraint. These
relative position constraints were created using odometry
differences.
Subsequently, to generate a map, the robot moves in a loop
to revisit the initial position. As the robot revisits a position it
already passed, the positions of the previously visited node
and the current node are compared to acquire the relative
position and this difference is utilized as a strong constraint.
The relative distance between previously visited node
and the current node is calculated using 2D LiDAR data
and the ICP algorithm. ICP algorithm aims to minimize
the difference between two clouds of points with the least
square method by iteratively minimizing relative position
differences until it reaches an optimized value.
Using these methods, the weak and strong constraints are
made completing the graph. Then, a non-linear solver called
Ceres [15] is utilized to find an optimized graph structure
that minimizes the least-square error. Finally, the 3D point
clouds at each optimized node will be added to a global 3D
map by process of transformation of perspective from robot
to global.
C. Wall Extraction
The planar geometries of buildings, walls, etc. are gen-
erally key features of robot mobility and representing them
with 3D point clouds take up a lot of memory and look
bumpy.
Since the ceiling and the floor are normally taken for
granted in an indoor environment and the existence of 3D
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on October 15,2021 at 00:38:56 UTC from IEEE Xplore. Restrictions apply.
data points representing them have low utility and take up
additional memory space, it is common to delete the ceiling
and floor from 3D data points.
However, the walls on the side are of much importance
(1) since it is required to assess the viable navigation paths
of a mobile robot. Also, since most walls are planar we
can extract them with filtering techniques to find the plane
equation of a wall and use it to our advantage to minimize
memory allocation. Additionally, projecting camera images
onto those extracted plane equations can give a smoother and
finer image of the planes.
(2) A plane wall can be extracted by using the following
method: Random Sample Consensus (RANSAC) [14], a
method to find a mathematical model, which, in this case,
is a plane, that best expresses data points with outliers, with
the following additional constraints.
1) Constraint 1: the plane equation’s normal should be
perpendicular to the z-axis.
(3) 2) Constraint 2: the plane area should be larger than a
threshold (e.g. 5 m2 ).
D. Image Projection onto the Extracted Wall
The primary method to color in the data points in the 3D
LiDAR map will be by projecting a single color image to an
extracted plane equation. Note that the color information of
the wall is best represented when the chosen image is taken
orthogonal to the wall. This method will theoretically be less
sparse and more smooth since we are creating more dense
and flat 3D data points, whereas the other data points are
sparse and uneven.
The following steps illustrates the steps taken to imple-
ment color projection from an image onto a segmented plane
equation.
1) Divide the point cloud data sets to two: the extracted
plane equation of
ax3 + by3 + cz3 + d = 0 (4)
and the remaining point cloud data set.
2) Produce a mask on top of the camera image which will
later discern the 3D points on the plane equation from
the rest of the 3D points.
a) Create a white background frame from an undis-
torted camera image
b) With the rest of the 3D points, find the corre-
sponding pixel coordinates at the white frame
and color it black. This will make the mask.
The corresponding pixel coordinates are obtained
using the following equation:
   
x2 1
x3
= fx fy (5)
y2 z3 y3
where (x3 , y3 , z3 ) are the 3D coordinates,
(x2 , y2 ) are the 2D coordinates on the image, and
(fx , fy ) are the focal lengths of the camera on
each axis direction.
3) Color the remaining 3D data points, those not on the
plane.
460
 (a)

Fig. 2: The white wall around the gray lockers is the

experiment target (semi-side view and semi-front view)

4) Now that the rest of the 3D points have color, we

need to imprint the segmented plane equation with the
camera image.
a) For this case, we produce 3D points on the plane
equation replacing the 3D points created through
a LiDAR. This is done by intersecting the plane
equation with the projection line from a camera (b)
image, again exploiting the camera matrix (5).
By replacing variables x3 and y3 in (4) with
x3 = xfx2 z3 and y3 = fyy2 z3 from (5), we get Fig. 3: (a) The camera image chosen which will be projected
 −1 onto the yellow plane in Fig. 4 and (b) the mask that blocks
by2
z3 = −d · ax fx
2
+ fy + c . Subsequently, the camera image from being projected onto 3D point clouds
x2 y2
x3 = fx z3 and y3 = fy z3 .
b) For each pixel in the camera if the corresponding
pixel in the mask is black, we do not produce
the new 3D points. However, if they are white,
we produce new 3D points and add the pixel’s
color information in that 3D point.

III. EXPERIMENT RESULTS & DISCUSSIONS

A. Robot Setup
As shown in Fig. 1, the robot used for this experiment
is a composition of a Yujin Robot Company product called
Turtlebot 2, a main PC (Intel NUC) with ROS installed, a
3D LiDAR (Velodyne Puck) installed in a slightly slanted
manner, and a camera (Jetson tx2 e-CAM) in the front.
Fig. 4: The extracted plane information are shown in varying
B. Indoor Environment
colors, out of which the yellow plane will be the area of
The experiment was carried out in an indoor environment. experimentation.
The images in Fig. 2 show the white wall which this
experiment focuses on.

461

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on October 15,2021 at 00:38:56 UTC from IEEE Xplore. Restrictions apply.
 C. Image Projection onto Plane Wall
The yellow area in Fig. 4 was extracted through RANSAC.
Following the steps of making a mask and filtering, not
allowing the dark regions to project into the plane equation,
we get the 3D point image as shown in Fig. 5 and Fig. 6,
which is a magnification of a small area of 5.
As shown in the two figures, we have successfully applied
pixels upon a plane making the data dense yet orderly and
smooth. Inspecting Fig. 6, which is a magnification of Fig.
5 at the boundary between the proposed method and the
original method, we notice that the regions on the right are
far denser than the region on the left.
A limitation of this technique is that the 3D plane can only
be as dense as the image projected onto it. Since the Jetson
tx2 e-CAM has a resolution of 640x480 pixels the resolution
of the wall can also only be that much. With further distance
from the plane the color accuracy can only get worse.
IV. CONCLUSION
This paper illustrates an attempt to manifest an extensively
useful 3D color map. The augmentation of 3D data points
in density and color information has been proven to be
only partially successful since the clean 3D color map
creation was restricted to flat surfaces. Despite the proposed
method’s limitations, the 3D color map region created with
an extracted plane equation boosts the aesthetic visualization
of the 3D map.
As mentioned at the end of the Experiment Results &
Discussion section, the resolution problem could be tack-
led by upgrading low resolution images to high resolution
through employing a higher resolution camera or Deep
Learning. Since this visualization process does not require
real-time performance, Deep Learning is a suitable candidate
to enhance images.
Although walls were relatively easy to imprint color
information since they are planes and images are also just
planes, placing accurate colors onto other objects, which are
not planes, presents complications.
One approach would be to segment 3D points of each
object into a mesh, which connects the data points to make
the outer surface of an object. Then coloring in the surfaces
represented as surface elements (surfels) would offer visually
denser 3D maps.
V. ACKNOWLEDGEMENTS
This work was supported by the Industry Core Technology
Development Project, 20005062, Development of Artificial
Intelligence Robot Autonomous Navigation Technology for
Agile Movement in Crowded Space, funded by the Ministry
of Trade, industry & Energy (MOTIE, Republic of Korea)
and in part by BK21+.
R EFERENCES
[1] C. Cadena et al., “Past, present, and future of simultaneous localization
and mapping: Toward the robust-perception age,” IEEE Trans. Robot.,
vol. 32, no. 6, pp. 1309–1332, Dec. 2016.
[2] D. Lee and H. Myung, “Solution to the SLAM Problem in Low
Dynamic Environments Using a Pose Graph and an RGB-D Sensor,”
Sensors, vol. 14, no. 7, pp. 12467–12496, Nov. 2014.
Fig. 5: Image projection onto the plane equation of the wall
on the right
Fig. 6: A magnified view of the boundary between the image
projection without (left) and with (right) a plane equation
[3] T. Oh, D. Lee, H. Kim, and H. Myung, “Graph Structure-Based
Simultaneous Localization and Mapping Using a Hybrid Method of
2D Laser Scan and Monocular Camera Image in Environments with
Laser Scan Ambiguity,” Sensors, vol. 15, no. 7, pp. 15830–15852,
Mar. 2015.
[4] H. Kim, D. Lee, T. Oh, H.-T. Choi, and H. Myung, “A Probabilistic
Feature Map-Based Localization System Using a Monocular Camera,”
Sensors, vol. 15, no. 9, pp. 21636–21659, 2015.
[5] J. Jung, S.-M. Lee, and H. Myung, “Indoor Mobile Robot Localization
and Mapping Based on Ambient Magnetic Fields and Aiding Radio
Sources,” IEEE Transactions on Instrumentation and Measurement,
vol. 64, no. 7, pp. 1922–1934, 2015.
[6] Y.-C. Lee and H. Myung, “Indoor Localization Method Based on
Sequential Motion Tracking Using Topological Path Map,” IEEE
Access, vol. 7, pp. 46187–46197, 2019.
[7] M. J. Milford, F. Schill, P. Corke, R. Mahony, and G. Wyeth, “Aerial
SLAM with a single camera using visual expectation,” 2011 IEEE
International Conference on Robotics and Automation, 2011.
[8] Lee, Donghwa, et al. “Vision-Based Object Detection and Tracking for
Autonomous Navigation of Underwater Robots.” Ocean Engineering,
vol. 48, 2012, pp. 59–68., doi:10.1016/j.oceaneng.2012.04.006.
[9] R. Guo, K. Peng, W. Fan, Y. Zhai, and Y. Liu, “RGB-D SLAM Using
Point–Plane Constraints for Indoor Environments,” Sensors, vol. 19,
no. 12, p. 2721, 2019.
[10] Hsiao, Ming, et al. “Keyframe-Based Dense Planar SLAM.” 2017
IEEE International Conference on Robotics and Automation (ICRA),
2017, doi:10.1109/icra.2017.7989597.
[11] Le, Phi-Hung, and Jana Kosecka. “Dense Piecewise Planar RGB-
D SLAM for Indoor Environments.” 2017 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), 2017,
doi:10.1109/iros.2017.8206375.
[12] V. D. Silva, J. Roche, and A. Kondoz, “Robust Fusion of LiDAR and
Wide-Angle Camera Data for Autonomous Mobile Robots,” Sensors,
vol. 18, no. 8, p. 2730, 2018.
[13] G. Grisetti, R. Kummerle, C. Stachniss, and W. Burgard, “A tutorial

462

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on October 15,2021 at 00:38:56 UTC from IEEE Xplore. Restrictions apply.
 on graph-based SLAM,” IEEE Intell. Transp. Syst. Mag., vol. 2, no.
4, pp. 31–43, Winter 2010.
[14] “How to use Random Sample Consensus model,” Docu-
mentation - Point Cloud Library (PCL). [Online]. Available:
http://pointclouds.org/documentation. [Accessed: 30-Jan-2020].
[15] “Ceres Solver,” Ceres Solver - A Large Scale Non-linear Optimization
Library. [Online]. Available: http://ceres-solver.org/. [Accessed: 30-
Jan-2020].

463

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY SURATHKAL. Downloaded on October 15,2021 at 00:38:56 UTC from IEEE Xplore. Restrictions apply.