Autonomous exploration through deep

reinforcement learning
Xiangda Yan, Jie Huang, Keyan He, Huajie Hong and Dasheng Xu
College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China

Abstract
Purpose – Robots equipped with LiDAR sensors can continuously perform efficient actions for mapping tasks to gradually build maps. However,
with the complexity and scale of the environment increasing, the computation cost is extremely steep. This study aims to propose a hybrid
autonomous exploration method that makes full use of LiDAR data, shortens the computation time in the decision-making process and improves
efficiency. The experiment proves that this method is feasible.
Design/methodology/approach – This study improves the mapping update module and proposes a full-mapping approach that fully exploits the
LiDAR data. Under the same hardware configuration conditions, the scope of the mapping is expanded, and the information obtained is increased.
In addition, a decision-making module based on reinforcement learning method is proposed, which can select the optimal or near-optimal
perceptual action by the learned policy. The decision-making module can shorten the computation time of the decision-making process and improve
the efficiency of decision-making.
Findings – The result shows that the hybrid autonomous exploration method offers good performance, which combines the learn-based policy with
traditional frontier-based policy.
Originality/value – This study proposes a hybrid autonomous exploration method, which combines the learn-based policy with traditional frontier-based
policy. Extensive experiment including real robots is conducted to evaluate the performance of the approach and proves that this method is feasible.
Keywords Mobile robots, Autonomous robots, Autonomous Exploration, Deep Q-network, Reinforcement learning
Paper type Research paper

1. Introduction et al., 2011), Cartographer (Hess et al., 2016) and Karto

Robot technology has developed rapidly in recent years.
Researchers have made great efforts and many attempts in
many fields. Now, robots can complete simple and repetitive
works very well, and even for works requiring high accuracy,
great progress has been made, such as robots on automatic
production lines (Qiu and Kermani, 2021). However, up to
now, the effect of robots performing tasks in complex
environments is still not very satisfactory. For example, we hope
that robots can replace humans to search, rescue and investigate
in harsh conditions, that is, robots can independently build
maps in unknown environments (Liu and Nejat, 2013).
Automatic mapping the unknown environments is a very
challenging task for robots. Scholars have done a lot of research
on autonomous mapping (Casper and Murphy, 2003). A
typical autonomous exploration system consists of a map-
updating module, a decision-making module, a path-planning
and motion-controlling module, as shown in Figure 1:

The map-updating module collects sensor data such
as LiDAR point cloud and depth camera data, then
updates the map based on sensor data. However, in the
current mapping algorithms proposed by scholars [such as
Gmapping (Grisetti et al., 2005), Hector (Kohlbrecher
(Konolige et al., 2010)], when the data is larger than the
effective range of the LiDAR, the grid probability of the
area within this range is not calculated (we call such
algorithms semi-mapping). We believe that this method
is reliable but inefficient, because the kind of algorithm
does not make full use of datas, and we have improved the
map-updating module on this basis.

The decision-making module determines the goal at the
next time according to the local observation map at the
current time and the robot’s information. A typical
scheme is to use the concept: frontier proposed by Brian
Yamauchi, extracting the area based on the junction of
open space and unknown space as the edge in the
occupancy grid map, and navigating a robot to the nearest
frontier to achieve the goal of autonomous exploration
(Yamauchi, 1997, 1998).

The path-planning and motion-controlling module plans
the path for the robot to reach the target, such as A  (Hart
et al., 1968), Hybrid A  (Dolgov and Thrun, 2009) and
RRT (LaValle and Kuffner, 2001), and controls the robot
to track the path to reach the target position, such as LQR
(Levinson et al., 2011), MPC (Nareyko et al., 2020) and
DWA (Fox et al., 1997). The algorithms all need to maintain

The current issue and full text archive of this journal is available on Emerald
Insight at: https://www.emerald.com/insight/0143-991X.htm The authors disclosed receipt of the following financial support for the
research, authorship and publication of this article: This research was
supported by NUDT, China.
Industrial Robot: the international journal of robotics research and application
50/5 (2023) 793–803 Received 19 October 2022
© Emerald Publishing Limited [ISSN 0143-991X] Revised 14 February 2023
[DOI 10.1108/IR-12-2022-0299] Accepted 27 February 2023

793
 Autonomous exploration
Xiangda Yan et al.
Figure 1 Autonomous exploration system
the global or local cost map in real time to prevent collisions,
which requires high computing resources for equipment.
This paper proposes a full-mapping method to improve the
map-updating module and trains the decision-making module
based on reinforcement learning to shorten the computation
time in the decision-making process and improve the efficiency
of decision-making.
Our paper proceeds as follows: we describe related work in
Section 2, elaborate on our methodology and agent network
design in Section 3, showcase experimental results in Section 4
and conclude in Section 5.
2. Related work
2.1 Information theory
There are many kinds of maps, such as the topology map, the
feature map and the grid map. But the feature map is usually
used in a structured environment.
A statistical approach derived from Bayesian theory has
captured our attention (Moravec, 1988). This method is fast
and effective, which considers the problem of generating a grid
map from independent sources of information and lays the
foundation for the mapping.
Thrun and Bücken (1996) describe an approach that
integrates two paradigms: grid-based and topological, and gives
results for autonomously operating a mobile robot equipped with
sonar sensors. However, the author assumes that all the walls are
perpendicular or parallel to each other, which is only suitable for
the environment of regular shape and has limitations.
Yamauchi (1997) proposed the concept of frontiers for the
first time. In the occupancy grid map, extract grids at the
junction of free grids and unknown grids in occupancy grid
maps as frontiers. And a robot navigates to the nearest frontier
to achieve the purpose of autonomous map building. Later,
Yamauchi (1998) extended this method to exploration for
multirobot. Robots share information and navigate to their
794
Industrial Robot: the international journal of robotics research and application
Volume 50 · Number 5 · 2023 · 793–803
nearest frontiers separately. However, no schedule has been
established, which results in low efficiency.
Simmons made some attempts to minimize the overall time
of the whole robot team, taking into account the cost of the
robot and the utility of the target point. And once the target
point is assigned to a robot, the utility of the target point to
other robots will be reduced. This approach reduces time
significantly, but there is still a relatively large optimization
space (Burgard et al., 2000; Simmons et al., 2000).
Laumond (1983) and Chatila and Laumond (1985) build a
topological model and then derive a semantic model, e.g.
identifying “rooms” and “corridors” (Kuipers and Byun,
1991). Nodes are used to represent specific locations in the
map, such as corners, and edges represent connections
between nodes, such as corridors. However, there is no use of
the topological model to cope with measure inaccuracy.
2.2 Reinforcement learning
At the same time, a variety of intelligent methods have emerged.
The ultimate goal of these methods is to minimize the time or the
path length for robots to explore the unknown environment.
It is the first time to try to solve the problem of exploring
unknown environments by adopting deep reinforcement
learning (Koutras et al., 2021). The author trains agents in an
Open-AI environment: Mars Explorer. Although the robot is a
brief model based on the particle in this environment, the size
of the environment is fixed and the environment is highly
structured, this is still a brave attempt in the exploration field
and has confirmed the possibility of using reinforcement
learning algorithms to solve this problem.
Zhang et al. (2021) and Chen et al. (2020) propose a
reinforcement learning method using a graph convolution
neural network to solve the decision-making problem in
exploration and train the policy in many random maps, taking
the historical position, current position, observation landmark
and candidate frontiers of the robot as the vertices of the graph
and connecting these vertices by some rules to form a graph.
However, this method still relies on a series of processing on
occupancy grid maps such as frontier detection to a certain
extent, and when the environmental characteristics change, the
efficiency of this graph-based method is not high. There is still a
lot of work to be done before deploying this method to robots.
Li et al. (2020) use a convolutional neural network (CNN)
to train agents and calculates the position with the largest
advantage value in the map by inputting the whole map into
the network. Finally, the author successfully deployed the
algorithm to robots and carried out experiments. However,
when the map size changes (i.e. the input dimension of the
neural network changes), the network needs to be changed and
retrained. What’s more, the author does not consider the
distance cost of robot movement, and the reward function is
only set to the map information gain obtained by the current
action (this is unreasonable).
3. Methodology
The algorithm is divided into three parts. In the first part, the
map-updating module is improved to make full use of sensor
data. In the second part, train the weight of the policy which
decides the goal (the path-planning and motion-controlling
 Autonomous exploration
Xiangda Yan et al.
module adopt the traditional A 1 DWA method). The third
part introduces map processing, frontier detection and
clustering. And the target calculation reinforcement learning
and mapping rescue algorithm are combined to do experiments
on the data set, and the feasibility and efficiency of the
algorithm are analyzed. The implementation details of each
part are described below:
3.1 Semi-mapping and full-mapping
There are many kinds of maps, considering that occupancy grid
map is easy to be applied in robot path planning, occupancy
grid map is adopted in this paper. The occupancy grid map
divides the whole map into grids. Each grid has only two states
(state: s). s = 1 means the grid is free, and s = 0 means the grid is
occupied. But in the real world, the information obtained by
robots from the external environment usually contains errors,
so it is more appropriate to use probability to represent the state
of the map. For a grid m in the map mi,j, use p(mi,j) to represent
the probability of the grid being occupied (called confidence
level bel(mi,j) and use pðm i;j Þ or 1  p(mi,j) to represent the
probability of the grid being idle. For the grid not observed by
the robot, its initial grid occupation probability p(mi,j) = 0.5
(Moravec, 1988).
The grid confidence level solution formula is as follows:


p mi;j jxt ; zt pðzt jxt Þbelt1 ðmi;j Þ
belt ðmi;j Þ ¼
pðmi;j Þpðzt jx1:t ; z1:t1 Þ
The following results are derived from Bayesian theory:
belt ðmi;j Þ ¼ 1  1=ð1 1 expðlt ÞÞ (2)
where:
8
>
> l ¼ lt1 1 linv  l0
> t
< l0 ¼ log  pðmi;j Þ=pðm i;j Þ
> to the robot is shown in Figure 3. We integrated two different
  (3)
>
> lt1 ¼ log belt1 ðmi;j Þ=belt1 ðm i;j Þ
>
: linv ¼ log p
mi;j jxt ; zt =p
m i;j jxt ; zt 
> neural network network (GRU) and the CNN. The output is
However, when the data is larger than the effective range of
the LiDAR, mapping algorithms do not calculate the grid
probability of the area within this range, such as Gmapping,
Hector, Cartographer and Karto (such algorithms called semi-
mapping). We believe that this kind of method is reliable but
inefficient because it does not make full use of data.
To solve this problem, we improved semi-mapping by using
all the valid data of LiDAR. When the LiDAR data is larger
than the effective range of the LiDAR, the grids in the limited
range can be considered to be free to achieve the purpose of
updating the map. Figure 2 shows the results of the two types of
mapping methods. The map updating area of the full-mapping
method is larger than that of the semi-mapping method, where
more information is obtained.
3.2 Deep reinforcement learning
Markov decision Process (MDP) can be used to model RL
problems. It is common to define an MDP as A quadruple
(S, A,r,f): S is the set of all environment states. st [ S represents
the state of the agent at time t; A is A set of actions that can be
executed by the agent. at [ A represents the action taken by the
agent at time t; r: S  A ! R is the reward function. rt  r (st, at)
Industrial Robot: the international journal of robotics research and application
Volume 50 · Number 5 · 2023 · 793–803
Figure 2 (a) Semi-mapping and (b) full-mapping
represents the immediate reward value obtained by the agent
when performing action at in st state. f: S  A  S ! [0,1] is the
state transition probability distribution function. st11  f(st,st)
represents the probability that the agent performs action at in
state st and transfers to the next state st11. The trajectory of
state, actions and rewards is t = (s0, a0, r0, s1, a1, r1, sT1, aT1,
rT1, sT), status, network update formula is:
 
Qðst ; at Þ 1 ¼ a Rt 1 1 1 gQðst 1 1 ; at 1 1 Þ  Qðst ; at Þ (4)
utarget ¼ rutarget 1 ð1  rÞu (5)
(1) Deep reinforcement learning is used to calculate goals. Because
the input dimension of the neural network is constant, the
entire map cannot be taken as input. We set the robot’s local
vision as state, that is, a grayscale image (marking the position
of the robot in this grayscale image), which size does not
change. And then scale the image to 84  84. The final result is
used as the input of the neural network. Action spaces are a
series of discrete actions, 81 actions in all. The position relative
neural networks in our framework, including the recurrent
influenced by the temporal sequential inputs to the neural
network, considering the temporal context of an input state.
What’s more, we add a dropout layer to avoid overfitting, as
shown in Figure 4.
Data set: We use the exploration data set to provide training
scenarios for the deep reinforcement learning method. The
exploration data set is highly diverse, with more than 10,000
maps in total. Figure 5 shows several 3D Gazebo simulation
environments.
Shannon entropy is used to represent the uncertainty of a
grid in the map, and entropy is introduced as the feature of the
grid (Carrillo et al., 2015).
Shannon entropy:
X



Hg   pðmi;j Þlogð pðmi;j ÞÞ 1 1  pðmi;j Þ log 1  pðmi;j Þ
ij
(6)
Reward function:
8
< R ¼ v1 R1 1 v2 R2 1 v3 R3
R1 ¼ DHg (7)
:
R2 ; R3 : path length; angle change
795
 Autonomous exploration
Xiangda Yan et al.
Figure 3 Action space
3.3 Frontier rescue
If the goal from deep reinforcement learning is not reachable or
the information obtained from the action is very low, frontier
rescue policy is carried out by clustering of frontiers based on
the Mean-Shift algorithm.
But frontiers directly extracted from occupancy grid
maps are decentralized. First, details are edge-detected and
denoising, using dilation and erosion at the same time, further,
the results are averaged. And make the obstacles expand to
ensure the accessibility of frontiers. Then, detect frontiers on
the result. On this basis, we detect the connectivity of frontiers
and group frontiers into different edges representing different
regions, which is shown in Algorithm 1.
Figure 4 Neural network architecture
796
Industrial Robot: the international journal of robotics research and application
Volume 50 · Number 5 · 2023 · 793–803
Figure 5 Exploration dataset
So far, the detected frontiers can be used to calculate the goal
for the robot (Senarathne et al., 2013). However, without
further processing, this process is cumbersome, inefficient and
prone to error. Therefore, Mean-Shift is performed on the
frontiers and uses the center of each class to calculate the goal
for the robot, which improves the efficiency of autonomous
exploration.
Mean-Shift is a hill-climbing algorithm that involves
shifting this kernel iteratively to a higher density region until
convergence. Every shift is defined by a mean shift vector. The
mean shift vector always points toward the direction of the
maximum increase in density. At every iteration, the kernel is
shifted to the centroid or the mean of the points within it. At
convergence, there will be no direction in which a shift can
accommodate more points inside the kernel. The bandwidth
size has a physical meaning, unlike k-means. Mean-Shift can
automatically determine the number of categories. Figure 6
shows the results of frontier rescue policy.
 Autonomous exploration Industrial Robot: the international journal of robotics research and application
Xiangda Yan et al. Volume 50 · Number 5 · 2023 · 793–803

Figure 6 Frontier detection and clustering Algorithm 1: frontiers grouping

1: Input: frontier list(Frontiers)

2: Output: clusters list(Clusters)
3: While Frontiers is not empty:
4: Initialize a list Clusters=[]
5: Initialize a list Open=[]
6: Initialize a list Close= []
7: Open append a frontier point,
Frontiers pop the point
8: Repeat:
9: If Open is not empty
10: Choose a frontier point from Open
11: Open append frontier points adjoining
with the point
12: Open pop the frontier point
13: Close append the frontier point
14: If Open is empty
15: Clusters append close
16: Until Open is empty

Source: Authors’ own work

Then, the greedy algorithm is used to calculate the exploration

scheme with the minimum cost for the robot. The path-
planning and motion-controlling module adopt the traditional
A 1 DWA method.

4. Experiments
The simulation experiment is carried out on Ros (the algorithm
is deployed to robots for experiments). To verify the effect of
the above algorithm, it is divided into two parts. In the first part,
DQN is trained to calculate the goal (in this process, the path-
planning and motion-controlling module adopt the traditional
A 1 DWA method). In the second part, experiments are done
on data sets by combining reinforcement learning and frontier
rescue, and the feasibility and efficiency of the algorithm are
analyzed. The experimental details of each part are described
below.

4.1 Autonomous exploration in the simulation

The training process is based on a virtual environment, ROS is
a systematic framework for robot development and Gazebo is
an open source 3D robot simulation software that integrates the
ODE physics engine, OpenGL rendering and supports sensor
simulation and actuator control.
The agent we trained was a standard robot development
platform from ROS and Gazebo: Scout-Mini. The Scout-Mini
is a robot platform that can be equipped with LiDAR sensors
and can be directly controlled via the speed topic, and strategies
trained on this standard robotic development platform can be
well transferable to other similar robotic platforms. The
minimum observation distance of the LiDAR sensor is 0.3 m,
and the maximum effective observation distance is 4 m. To be
closer to the real environment, Gaussian noise is added to the
sensor data. The raw data from the LiDAR sensor is sampled at
a range of 360 degrees.
We trained the strategy in Gazebo 3D environments similar
to a tunnel or corridor, using about 5,000 maps during the

797
 Autonomous exploration
Xiangda Yan et al.
training process. Maps are of unequal size. In our experiment,
the robot’s LiDAR provided sensor data with a 360-degree field
of view with Gaussian noise. The local observation map of
the robot is used as the input state quantity of the neural
network, and the position of the robot is marked in it (its gray
value is set to 80 within the radius of 0.3 m). Detect if a collision
would take place or not before the robot takes any action, and
training samples will include collision. Then, the robot will
gradually learn the strategy of avoiding the collision and
exploring at the same time. However, if the robot navigates to
the end of the channel or cannot find an effective action, the
“Frontier Rescue” mechanism will be triggered to drive the
robot to the nearest boundary point so that the training can
continue.
We compared RRT-Exploration (Umari and
Mukhopadhyay, 2017), Explore-lite (Hörner, 2016) and our
learned policy in detail on a data set consisting of an additional
five maps at the stage of testing. RRT-Exploration presents a
new strategy for detecting frontier points using RRT. A
growing randomly sampling tree is used in the search for
frontier points. Explore-lite is a method to search frontiers
points by breadth-first search on the map. The frontier
points are filtered and queued to be assigned to a robot, then
the robot moves toward the assigned point. Both algorithms
detect the frontiers first, while DRL directly gets the goal
through the local field of view, which can more obviously show
the end-to-end characteristics and highlight the efficiency of
our algorithm. In the process of exploration based on DQN,
any action that will result in low Shannon entropy gain will
trigger the “Frontier Rescue” mechanism. It can guide the
robot from a region full of obstacles or with poor information to
the nearest frontier point by introducing this mechanism.
In the training process, only when there are no frontier
points on the map or the exploration proportion is up to
the threshold, the exploration is stopped. We choose two
performance indexes to compare performance. Reward
represents the area of the explored region or the information
gain obtained during exploration, which is the key index
to evaluate the performance of a method (excluding the
reward obtained by the “Frontier Rescue” mechanism). If the
rewards are the same, the path length of the robot is also a key
index.
We used the DQN algorithm to train the network and set the
experience playback mechanism to improve the efficiency of
using samples. Before starting the training, the experience
playback pool should collect at least 10,000 samples, and the
maximum number of samples in the experience playback pool
is 50,000. The network is trained every 20 steps, and 64
samples are randomly sampled from the experience replay pool
for network gradient update at a time.
The learning rate is set to 0.0001, the sample batch is set to
64, the reward discount rate is set to 0.99, the network soft
update rate is set to 0.01 and the network soft update is
performed every 250 training times. Train the model on a
device configured with 32GB RAM, CPU: i7 11800H and
Nvidia 3080 6GB GPU. We trained 50,000 episodes on the
data set and spent 20 h training the network.
We successfully used the DQN algorithm to train the agent to
complete the end-to-end decision-making process in a complex
and diverse environment. What’s more, we trained six different
798
Industrial Robot: the international journal of robotics research and application
Volume 50 · Number 5 · 2023 · 793–803
strategies and recorded the final average reward of different
strategies. Figure 7(a) – (f) shows the corresponding six action
spaces, and the average rewards corresponding to the six
strategies are 3.897, 3.267, 4.217, 3.937, 4.015 and 3.917,
respectively, as shown in Figure 7(g). When the action space is
Figure 7(c), the final average reward after convergence is 4.217,
the highest among the six strategies. We study the relationship
between quantity and distribution of action space and the
advantages and disadvantages of learned policies. We may
safely draw a conclusion from Figure 7 that (1) if the number
of actions is the same, but the distribution is different, it all
can learn policies that explore unknown environments.
However, the final reward of action space (c) in Figure 7 will
 
be slightly higher by about 5% ð4:2174:015
4:015
Þ
 100% than the
other five strategies; (2) if the distribution of actions is the
same, but the number is different, it all can learn policies
that explore unknown environments. However, note that the
more actions are used, the higher the final reward would be.
Because if there are too few or too many actions, it is all difficult
to learn a good policy. Ultimately, we adopt action space (c) in
Figure 7.
Figure 8 shows the change curve of loss and reward in the
training process. When the agent learns a good policy, loss
decrease gradually. What’s more, in the initial stage, the agent is
constantly exploring, and the reward is unstable. After the agent
learns a policy that can obtain a large reward, the reward starts
to increase gradually, and gradually converges after 35,000
episodes.
What’s more, we compared the performances of the
proposed algorithm with RRT-Exploration and Explore-lite,
conducted experiments using these exploration methods on
five experimental scenarios and recorded the data of these
exploration methods. We also set up a group of experiments in
which human operate robots remotely to explore as a baseline.
All methods can explore more than 98% of the area in the
unknown environment. Figure 9 shows an example of using
these exploration methods to explore an unknown environment
[the action space is (c) in Figure 7] when exploring Scenario 1.
In the whole process of autonomous exploration, fully rely on
DQN to make decisions. According to Figure 10, the shortest
exploration route obtained from the four methods is controlled
by human. Among the three algorithms, our method is that the
exploration route is optimal. And by comparing the time cost of
the algorithm, our algorithm consumes about 2 ms for every
decision, which greatly improves the real-time performance.
From the performance comparison of the three algorithms, we
can roughly see the advantages of the DQN method in path
exploration efficiency and decision time compared with RRT
method and LITE method. Therefore, it can be concluded that
our method can improve the efficiency, comparing with RRT-
Exploration and Explore-lite.
What’s more, because both RRT-Exploration and Explore-
lite need to detect frontiers on the global map, detection
efficiency is closely related to map size and complexity of the
scenario. So, the two methods have large and significantly
positive correlation differences in decision-making time with
different complexity. However, DQN method only needs to
obtain the local field of view without any further processing.
Then, the goal can be determined by inputting local map into
 Autonomous exploration Industrial Robot: the international journal of robotics research and application
Xiangda Yan et al. Volume 50 · Number 5 · 2023 · 793–803

Figure 7 Six action spaces

Figure 8 DQN learning curves

799
 Autonomous exploration Industrial Robot: the international journal of robotics research and application
Xiangda Yan et al. Volume 50 · Number 5 · 2023 · 793–803

Figure 9 Exploration pathway Figure 10 Performance comparison

Exploration path length( ) Decision-making time

Scenario Efficiency ( )
HM DQN RRT LITE DQN RRT LITE

95.5 106.5 119.8 113.5

2.0 134.2 41.3
1.0 89.6% 79.6% 84.1%

60.1 66.1 67.1 66.3

2.0 128.1 41.1
1.0 90.9% 89.4% 90.6%

46.2 46.9 68.1 49.6

1.9 120.5 33.6
1.0 98.5% 67.9% 93.1%

67.4 69.6 70.2 74.7

2.0 100.8 37.5
1.0 96.7% 95.9% 90.2%

44.4 44.5 44.8 45.7

1.9 114.4 31.8
1.0 99.7% 99.0% 97.1%

Notes: Efficiency is dividing exploration path length of the

exploration method into exploration path length of human
Source: Authors’ own work

viewed online: https://www.bilibili.com/video/BV1Q8411e7WL

the neural network, and decision time is not related to map size
and environmental complexity. So decision time of DQN
method is almost irrelevant to the complexity of the scenario
and almost remains constant.
4.2 Hybrid autonomous exploration experiment
In most cases, the above algorithm can be used for mapping.
We tested on the test set, and our method can explore more
than 95% of the area on all maps. However, because of the
intricacy and the multiplicity of the world, if the goal is
unreachable or the information obtained from the action is
very low, or even if the robot collided with obstacles, frontier
detection and clustering should be used to calculate the goal for
robot rescue. This method can greatly improve the robustness
and reliability of the algorithm. We propose a hybrid autonomous
exploration method that combines the learn-based policy with the
frontier rescue policy.
Extensive experiments including real robots are conducted to
evaluate the performance of the approach. Figure 11 proves
that our method is feasible in the real world. Figure 11(a) shows
the real robot we used in the demonstrations. The motion
platform is Scout-Mini, equipped with VLP 16 laser radar,
and provides locating information through the odometer and
IMU on the vehicle. The communication between equipment
is supported through the router, and the Jetson Nano is
used as the master control device to process data and run
our algorithm. We tested the hybrid autonomous exploration
algorithm in two realistic environments. Figure 11(b) shows
the real experimental scene similar to a warehouse, baffles in the
environment greatly increase the complexity of the environment
and improve the difficulty of autonomous exploration. In
addition, we tested the performance of the algorithm in another
larger building, as shown in Figure 11(c).
Figure 12 shows the changes at different times in the
exploration process, two videos of the experiments can be
and https://www.bilibili.com/video/BV1CW4y1j7ey. At any
time, the robot calculates the goal through the policy learned
by deep reinforcement learning. When the goal cannot be
reached or the information gain is less than the minimum
expectation set by us, the “Frontier Rescue” mechanism is
started. This mechanism can guide the robot out of the area
where information is scarce and avoid system disorders.
Figure 12 shows the occupancy grid obtained during completing
the autonomous exploration task on the real robot. By comparing
our experimental environment with the final results obtained
through hybrid autonomous exploration, it can be found that
the environmental features are completely established. Our
method explores more than 95% of the area in unknown
environments and realizes the autonomous exploration task of an
unknown real environment without any prior knowledge.
5. Conclusion
We improved the map-updating module and proposed a
full-mapping method and a reinforcement learning method to
train the decision-making module. The learned policy can
select the optimal or near-optimal perceptual action, and it
has been demonstrated through experiments that our method
is feasible. In most cases, the above algorithms can be used
for mapping. However, sometimes, the goal of the policy is
unreachable or the information obtained by the action is very
low. We propose a hybrid autonomous exploration method that
combines the learn-based policy with the frontier rescue policy.
This method greatly improves the robustness and reliability of
the algorithm. The proposed method learns policy on more
than 5,000 training map data sets and is tested by performing a
total of five exploration runs. Results obtained show that the
proposed strategy can successfully explore the entire map in a
reasonable amount of path length and without substantially
losing performance when compared to RRT-exploration and
Explore-lite. Future work will consider cooperative exploration
for multirobot.

800
 Autonomous exploration Industrial Robot: the international journal of robotics research and application
Xiangda Yan et al. Volume 50 · Number 5 · 2023 · 793–803

Figure 11 Real robot and scenario

Figure 12 Hybrid autonomous exploration in real robot

801
 Autonomous exploration
Xiangda Yan et al.
References
Burgard, W., Moors, M., Fox, D., Simmons, R. and Thrun, S.
(2000), “Collaborative multi-robot exploration”, IEEE
International Conference on Robotics and Automation, Vol. 1,
pp. 476-481, doi: 10.1109/ROBOT.2000.844100.
Carrillo, H., Dames, P., Kumar, V. and Castellanos, J.A.
(2015), “Autonomous robotic exploration using occupancy
grid maps and graph SLAM based on Shannon and
Renyi entropy”, IEEE International Conference on Robotics
and Automation, pp. 487-494, doi: 10.1109/ICRA.2015.
7139224.
Casper, J. and Murphy, R. (2003), “Human-robot interactions
during the robot-assisted urban search and rescue response
at the World Trade Center”, IEEE Transactions on Systems,
Man, and Cybernetics, Part B (Cybernetics), Vol. 33 No. 3,
pp. 367-385, doi: 10.1007/s10846-013-9822-x.
Chatila, R. and Laumond, J.P.P. (1985), “Position referencing
and consistent world modeling for mobile robots”, IEEE
International Conference on Robotics and Automation,
pp. 138-145, doi: 10.1109/robot.1985.1087373.
Chen, F., Martin, J.D., Huang, Y., Wang, J. and Englot, B.
(2020), “Autonomous exploration under uncertainty via
deep reinforcement learning on graphs”, IEEE/RSJ
International Conference on Intelligent Robots and Systems,
pp. 6140-6147, doi: 10.1109/IROS45743.2020.9341657.
Dolgov, D. and Thrun, S. (2009), “Autonomous driving in
semi-structured environments: mapping and planning”,
IEEE International Conference on Robotics and Automation,
pp. 3407-3414, doi: 10.1109/ROBOT.2009.5152682.
Fox, D., Burgard, W. and Thrun, S. (1997), “The dynamic
window approach to collision avoidance”, IEEE Robotics &
Automation Magazine, Vol. 4 No. 1, pp. 23-33, doi: 10.1109/
100.580977.
Grisetti, G., Stachniss, C. and Burgard, W. (2005), “Improving
grid-based SLAM with Rao-Blackwellized particle filters
by adaptive proposals and selective resampling”, IEEE
International Conference on Robotics and Automation,
pp. 2432-2437, doi: 10.1109/ROBOT.2005.1570477.
Hart, P.E., Nilsson, N.J. and Raphael, B. (1968), “A
formal basis for the heuristic determination of minimum
cost paths”, IEEE Transactions on Systems Science and
Cybernetics, Vol. 4 No. 2, pp. 100-107, doi: 10.1109/TSSC.
1968.300136.
Hess, W., Kohler, D., Rapp, H. and Andor, D. (2016), “Real-
time loop closure in 2D LIDAR SLAM”, IEEE International
Conference on Robotics and Automation, pp. 1271-1278,
doi: 10.1109/ICRA.2016.7487258.
Hörner, J. (2016), “Map-merging for multi-robot system”,
Charles University in Prague, Faculty of Mathematics and
Physics, Prague, available at: https://is.cuni.cz/webapps/zzp/
detail/174125/
Kohlbrecher, S., von Stryk, O., Meyer, J. and Klingauf, U.
(2011), “A flexible and scalable SLAM system with full 3D
motion estimation”, IEEE International Symposium on Safety,
Security, and Rescue Robotics, pp. 155-160, doi: 10.1109/
SSRR.2011.6106777.
Konolige, K., Grisetti, G., Kümmerle, R., Burgard, W.,
Limketkai, B. and Vincent, R. (2010), “Efficient sparse pose
adjustment for 2D mapping”, IEEE/RSJ International
802
Industrial Robot: the international journal of robotics research and application
Volume 50 · Number 5 · 2023 · 793–803
Conference on Intelligent Robots and Systems, pp. 22-29, doi:
10.1109/IROS.2010.5649043.
Koutras, D.I., Kapoutsis, A.C., Amanatiadis, A.A. and
Kosmatopoulos, E.B. (2021), “MarsExplorer: exploration
of unknown terrains via deep reinforcement learning and
procedurally generated environments”, arXiv preprint arXiv:
2107.09996.
Kuipers, B. and Byun, Y.T. (1991), “A robot exploration and
mapping strategy based on a semantic hierarchy of spatial
representations”, Robotics and Autonomous Systems, Vol. 8
Nos 1/2, pp. 47-63.
Laumond, J.P.P. (1983), “Model structuring and concept
recognition: two aspects of learning for a mobile robot”,
International Joint Conference on Artificial Intelligence,
pp. 839-841.
LaValle, S.M. and Kuffner, J.J. (2001), “Randomized
kinodynamic planning”, The International Journal of Robotics
Research, Vol. 20 No. 5, pp. 378-400, doi: 10.1177/
02783640122067453.
Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D.,
Kammel, S., Kolter, J.Z., Langer, D., Pink, O., Pratt, V.
and Sokolsky, M. (2011), “Towards fully autonomous
driving: systems and algorithms”, IEEE Intelligent Vehicles
Symposium (IV), pp. 163-168, doi: 10.1109/IVS.2011.
5940562.
Li, H., Zhang, Q. and Zhao, D. (2020), “Deep reinforcement
learning-based automatic exploration for navigation in
unknown environment”, IEEE Transactions on Neural
Networks and Learning Systems, Vol. 31 No. 6,
pp. 2064-2076, doi: 10.1109/TNNLS.2019.2927869.
Liu, Y. and Nejat, G. (2013), “Robotic urban search and
rescue: a survey from the control perspective”, Journal of
Intelligent & Robotic Systems, Vol. 72 No. 2, pp. 147-165,
doi: 10.1007/s10846-013-9822-x.
Moravec, H.P. (1988), “Sensor fusion in certainty grids for
mobile robots”, AI Magazine, Vol. 9 No. 2, pp. 61-74,
doi: 10.5555/46184.46187.
Nareyko, G., Biemelt, P. and Trächtler, A. (2020), “Real-time
optimized model predictive control of an active roll
stabilization system with actuator limitations”, IFAC-
PapersOnLine, Vol. 53 No. 2, pp. 14375-14380, doi:
10.1016/j.ifacol.2020.12.1393.
Qiu, S. and Kermani, M.R. (2021), “Inverse kinematics of high
dimensional robotic arm-hand systems for precision
grasping”, Journal of Intelligent & Robotic Systems, Vol. 101
No. 4, pp. 1-15, doi: 10.1007/s10846-021-01349-7.
Senarathne, P., Wang, D., Wang, Z. and Chen, Q. (2013),
“Efficient frontier detection and management for robot
exploration”, IEEE International Conference on Cyber
Technology in Automation, Control and Intelligent Systems,
pp. 114-119, doi: 10.1109/CYBER.2013.6705430.
Simmons, R., Apfelbaum, D., Burgard, W., Fox, D., Moors,
M., Thrun, S. and Younes, H. (2000), “Coordination for
multi-robot exploration and mapping”, The National
Conference on Artificial Intelligence, pp. 852-858.
Thrun, S.B. and Bücken, A. (1996), “Integrating grid-based
and topological maps for mobile robot navigation”, The
National Conference on Artificial Intelligence, pp. 944-950.
Umari, H. and Mukhopadhyay, S. (2017), “Autonomous
robotic exploration based on multiple rapidly-exploring
 Autonomous exploration
Xiangda Yan et al.
randomized trees”, IEEE/RSJ International Conference on
Intelligent Robots and Systems, pp. 1396-1402, doi:
10.1109/IROS.2017.8202319.
Yamauchi, B. (1997), “A frontier-based approach for
autonomous exploration”, IEEE International
Symposium on Computer Intelligence in Robotics and
Automation, pp. 146-151, doi: 10.1109/CIRA.1997.
613851.
Yamauchi, B. (1998), “Frontier-based exploration using
multiple robots”, IEEE International Conference on
For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: permissions@emeraldinsight.com
803
Industrial Robot: the international journal of robotics research and application
Volume 50 · Number 5 · 2023 · 793–803
Autonomous Agents, pp. 3715-3720, doi: 10.1145/280765.
280773.
Zhang, Z., Shi, C., Zhu, P., Zeng, Z. and Zhang, H.
(2021), “Autonomous exploration of mobile robots via
deep reinforcement learning based on spatiotemporal
information on graph”, Applied Sciences, Vol. 11 No. 18,
pp. 8299-8320.
Corresponding author
Keyan He can be contacted at: hekeyan2009@126.com