Neural Computing and Applications (2024) 36:4523–4535

https://doi.org/10.1007/s00521-023-09396-x (0123456789().,-volV)(0123456789().,-volV)

REVIEW

Improving productivity in mining operations: a deep reinforcement

learning model for effective material supply and equipment
management
Teddy V. Chiarot Villegas1 • S. Francisco Segura Altamirano1 • Diana M. Castro Cárdenas1 •

Ayax M. Sifuentes Montes2 • Lucia I. Chaman Cabrera1 • Antenor S. Aliaga Zegarra2 •

Carlos L. Oblitas Vera1 • José C. Alban Palacios2

Received: 24 May 2023 / Accepted: 13 December 2023 / Published online: 13 January 2024
 The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024

Abstract
This research study examines the impact of shifts and lunch breaks on mining operations, particularly focusing on delays in
hauling equipment used to supply extracted material to crushers. These delays significantly reduce productivity, averaging
below 80% during regular working hours and adversely affecting mine profitability. To address this issue, a Q-learning-
based deep reinforcement learning model was developed, utilizing real-world data from mining operations. The model
aimed to achieve a 90% coverage in material supply to the crushers. A simulation environment, closely resembling the
physical mining setting, was created to test the trucks as agents. Various scenarios, including equipment selection, cycle
time, queue times, and material types, were considered. Based on the results, a deep learning model was trained to
maximize coverage by determining the optimal combination of trucks and crushers. The solution successfully achieved a
90% supply coverage during shift changes and lunch breaks, with average execution times of less than 1 ms, making it
suitable for real-time applications. This research demonstrates the effectiveness of the proposed Q-learning deep rein-
forcement learning model in optimizing material supply and enhancing mining productivity. By addressing delays and
improving operational efficiency, this model holds significant potential for improving profitability in mining operations.

Keywords Mining operations  Hauling equipment  Q-learning  Truck dispatching

1 Introduction shovels presents significant challenges in achieving daily

Efficient fleet management is crucial in mining operations
to maximize productivity and minimize costs, which con-
stitute approximately 50% of total operational expenses in
open-pit mines. The scarcity of hauling trucks and loading
tonnage objectives. Such delays hinder the capacity to clear
mining fronts for progression or to supply feed for crushing
processes [1].
Efficient resource management plays a pivotal role in
addressing the aforementioned challenges, and many

Antenor S. Aliaga Zegarra

& S. Francisco Segura Altamirano aaliagaz@unp.edu.pe
sseguraal@unprg.edu.pe
Carlos L. Oblitas Vera
Teddy V. Chiarot Villegas coblitas@unprg.edu.pe
tchiarot@unprg.edu.pe
José C. Alban Palacios
Diana M. Castro Cárdenas jalbanp@unp.edu.pe
dcastroc@unprg.edu.pe
1
Ayax M. Sifuentes Montes Pedro Ruiz Gallo National University, Calle Chile 848, JLO,
asifuentesm@unp.edu.pe Chiclayo, Lambayeque, Peru
2
Lucia I. Chaman Cabrera National University of Piura, Piura, Peru
lchamanc@unprg.edu.pe

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

4524
mining activities employ fleet management systems to
maximize production, minimize stockpile usage, align
crusher feeding with objectives, and adhere to blending
restrictions. However, these systems offer multiple con-
figuration options determined by the operator’s criteria and
experience. Notably, the utilization and prioritization of
shovels and crushers hold immense significance within this
context. Dispatch decisions carry substantial weight in
operational efficiency and are of utmost importance, con-
sidering that a significant portion of mining costs are
directly associated with truck and shovel activities [2].
Many companies offer mining fleet management sys-
tems and the most used ones are Modular Mining Systems,
Jigsaw Software, and Wenco, installed and used by 200,
130, and 65 mining companies, respectively. However,
Micro mines with the Pitram system and Caterpillar with
Cat MineStar Solutions are the next leaders in mining fleet
management systems [3].
Recent research has explored innovative solutions, such
as integrating control layers into fleet management systems
using deep learning to improve dispatch agility [4]. Such
intelligent distributed architectures, namely a vehicular
layer, a processing layer and a decision layer, with high-
performance algorithms and Internet-of-Things (IoT) con-
nectivity have demonstrated potential for enhanced real-
time coordination. Experimental implementations show-
case scalability possibilities across various mines.
Another study [5] utilized dense neural network models
in open-pit mining truck systems to predict daily mined
material production based on transport conditions and
operating times. The verification with data processed over
two months showed that the ore production in the morning
and afternoon were 11.40% and 8.87%, respectively, with
an error margin of approximately 4.17% between actual
and forecasted productions. This method of prediction can
effectively address the challenges faced by conventional
truck system simulations based on complex algorithms.
Further studies have applied deep multi-agent rein-
forcement learning (RL) for vehicle supply-demand bal-
ance in large-scale ride-sharing platforms, coordinating
thousands of units to enhance revenue and customer sat-
isfaction [6]. Adapting such approaches to optimize shovel-
truck fleets could transform mining productivity. Our work
employs Q-learning-based models with tailored rewards to
significantly improve crusher coverage. Rigorous testing
on extensive data from Peru analyzes truck agent behavior
in shovel selection. By prioritizing ore movement and
penalizing waste, the networks achieved significant
improvements in efficiency and resource utilization. Fur-
ther leveraging the latest advancements in ride-sharing
could offer a highly promising transformation in fleet
management.
123
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
Neural Computing and Applications (2024) 36:4523–4535
A recent advancement in the area of large-scale ride-
sharing platforms is the balancing of vehicle supply and
demand, key to maximizing efficiency. A study by Lin
et al. developed new contextual deep multi-agent rein-
forcement learning (RL) algorithms that effectively coor-
dinated thousands of vehicles to improve platform revenue
and customer satisfaction [7].
Building on this, [8] presents an actor-critic RL model to
provide updated short-term production schedules and fleet
assignments for trucks and shovels in a mining complex.
The carefully designed reward system prioritizes mineral
material movement and penalizes tailings. Extensive sim-
ulation demonstrates significantly enhanced productivity,
efficiency, and resource utilization compared to current
practices. By harnessing recent innovations in ride-sharing
and mining through advanced machine learning, this study
offers a promising direction for transforming fleet
management.
Ahangaran et al. [9] explored the truck assignment
problem in open-pit mines, aiming to maximize produc-
tivity while minimizing transportation costs. Their two-
stage model first determines optimal routes using flow
network theory and then assigns trucks to shovels using
binary integer programming, accommodating operational
constraints and varying truck capacities. Mu et al. [10]
addressed the efficiency challenges in dispatching large
truck fleets, given the complexity of current road networks.
They proposed a multilayer map matching algorithm with
spatial data structures and a novel ‘‘lookup table’’ method,
outperforming traditional ‘‘pinpointing’’ techniques in
empirical tests with Taiwanese truck data.
Chaowasakoo et al. [11] studied truck dispatch strategies
in open-pit mining to maximize productivity under cycle
time uncertainties. Their simulation comparing four
strategies revealed that the global vision approach signifi-
cantly outperformed traditional methods in a real coal mine
scenario.
In summary, the reviewed studies effectively employ
various analytical and simulation techniques, including
integer programming, goal programming, hidden Markov
models, and reinforcement learning algorithms to optimize
complex real-world transport and mining systems. Their
findings demonstrate significant quantitative improve-
ments, ranging from 47 to 270% gains in key performance
indicators such as production levels, cash flows, truck
delays and fleet availability. These underscore outstanding
yet realistic opportunities for operational advancement
leveraging modern optimization, machine learning and
data-driven solutions.
Addressing the persistent challenge of production bot-
tlenecks in mining operations, an emergent approach in the
field involves integrating an automated control layer into
fleet management systems, harnessing the optimization
Neural Computing and Applications (2024) 36:4523–4535
power of deep reinforcement learning models. Aligning
with this high-potential direction, in this study we present a
breakthrough proposition utilizing a deep Q-learning
algorithm tailored to the specifics of the mining context.
Our proposed model demonstrates remarkable potential to
significantly enhance truck coverage and material supply to
crushers during shift changes and lunch breaks. This
unprecedented achievement is realized through meticulous
and constraints-aware reward design, dynamically ensuring
optimal fleet performance in varying conditions. By
effectively harnessing the generalization capability of deep
learning, our research puts forward a game-changing
solution poised to revolutionize the mining industry,
overcoming intrinsic production limitations and catalyzing
the unlocking of untapped operational efficiency at scale.
2 Materials and methods
Machine learning, a branch of artificial intelligence that
enables computers to learn without being explicitly pro-
grammed. Through algorithms, patterns are identified in
data and prediction or classification models are con-
structed. There are three types of learning: supervised,
reinforcement, and unsupervised. Supervised learning uses
labeled data to train a model and establish an input/output
relationship. In unsupervised learning, data is unlabeled,
and the algorithm must classify the information. In this
study, reinforcement learning is prioritized, where a model
is developed with an agent that interacts in an unknown
environment and learns through trial and error, using
established rewards and punishments. By employing these
machine learning techniques, this study pushes the
boundaries of knowledge and explores the potential for
data-driven advancements, fostering innovation in various
areas (see Fig. 1).
The terms shown in Fig. 1 can be described as:
Environment in this context is characterized by rewards
obtained from the environment itself, actions performed by
the agent or input to the environment, and observations that
provide additional information on the rewards acquired by
the agent.
Reward are quantitative values obtained from the
environment, serving as feedback to the agent. They can be
positive or negative, influencing the agent’s behavior and
learning process.
Action This represents the agent’s behavior within the
environment, which can be accepted by the game or
activities in progress. In reinforcement learning, actions
can be classified as discrete or continuous. Discrete actions
are a finite set of options, enabling the agent to move
between specific choices, such as left or right. On the other
hand, continuous actions involve values associated with the
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
4525
Fig. 1 Reinforcement learning model
action itself, such as the steering wheel actions used in
driving cars, which can vary in different directions and
steering angles.
Observations serve as a reference provided by the
environment to the agent, offering insights into the sur-
rounding circumstances. This information is crucial for
determining rewards, as it includes data on rewards, pun-
ishments, and the agent’s acquired score, facilitating the
learning process.
2.1 Q learning
Part of the reinforcement learning methodology solves the
learning difficulties in the control of autonomous agents
through trial and error within a dynamic environment, by
providing a signal that allows each of the actions carried
out to be improved. Furthermore, it can solve decision
problems in which the benefit of an action is subject to a
chain of Markovian decisions. In real time, the agent is in a
particular position S0 of in the environment and then per-
forms an action a0 generating a new state S1 and reward r 0 ,
and so on, as shown in Fig. 2. The agent performs a series
of actions whose values are defined by the reward function
over time, and the agent decides what action to take during
each step to maximize the value of the reward. Generally,
the environment and the task without prior external influ-
ences are considered, such that they satisfy the Markovian
property. A set of possible states S, actions A, transition
probabilities Pðs; a; s0Þ, expected rewards Rðs; a; s0Þ, and
cumulative reward Rt are added in the future, as shown in
Eq. 1, where c is a discount factor whose value ranges
from 0 to 1 [12].
Fig. 2 Q-learning methodology
123
4526
8
> S ¼ s0 ; s1 ; s2 ;    ; sn
>
>
>
< A ¼ a0 ; a1 ; a2 ;    ; an
Pðs; a; s0Þ ¼ Pfstþ1 ¼ s0; st ¼ s; at ¼ ag ð1Þ
>
> 2
> Rðs; a;Ps0Þ ¼ Efr tþ1 ; st ¼ s; at ¼ a; stþ1 ¼ s0g
> ponents [15]:
: 1
Rt ¼ k¼0 ck r tþkþ1
Control theory methods are used in each episode and
algorithms capable of making better predictions, whose
value a [ (0,1) updates the values of Qðs; aÞ (see Eq. 2) are
applied. It is reduced step by step or constantly, depending
on whether or not the work is in a static environment [12]
Qðs; aÞ ¼ Qðs; aÞ þ a½r þ rfmaxg Qðs0; a0Þ  Qðs; aÞ ð2Þ
Reinforcement learning uses specific criteria, which are
detailed below [13]:
• Defines multiple episodes, which record loss values in a
list;
• Sets random actions under some policy, for example,
the direction of a path to be forward, backward, left, or
right (see Fig. 2);
• Each action a carries a reward r, and a new state stþ1 is
generated;
• Collects the transition ðs; a; r; s0Þ in the experience
storage list (see Fig. 3), providing batch updates online,
where each experience is generated based on probabil-
ity, taking e as the value for designated actions and then
saving them in a list. Therefore, it decomposes into
separate mini-batch tensors, iterates over them, and
computes updates for the mini-batch Q-values. Then,
the final array Y is aggregated, and each state is stored
in variable X to form a mini training batch [14].
• Determines the gradient descent loss function which
decreases the neural network’s prediction error.
2.2 Mining fleet management system
There are several large-scale mining management software
packages, including DISPATCH, with data
Fig. 3 Probability strategy
123
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
Neural Computing and Applications (2024) 36:4523–4535
communication functions, GPS global positioning sensors
for shovels, drills, and tractors, and automatic assignments
to haul trucks in open-pit mines, with the following com-
• PTX (Modular Mining) computerized field system
comprising a graphic console and a central unit;
installed on trucks, auxiliary equipment, shovels, and
crushers;
• Global positioning system (GPS) technology;
• The radio data link is a base computer in the
information center, whose purpose is to allocate auto-
matic assignments to haul trucks.
The hauling process begins at the beginning of the shift,
where each operator data are entered in the Dispatch sys-
tem for the subsequent registration of both the user and the
truck, shovel, or crusher. The data are sent to the infor-
mation center after confirming the registration. Then, the
operator receives a message regarding the necessity for the
continuous use of equipment and the mandatory revisions,
after which the operator and the equipment can continue
the hauling cycle. For breaks such as operator replace-
ments, shift changes, maintenance, refueling, and blasting
processes that occur in mines, Dispatch performs the
allocation and optimization calculations before operators
are entered into the system.
3 Analysis and proposal
Currently, mining companies have hundreds of trucks,
dozens of shovels, several loaders, and primary crushers
distributed among different pits. A probable scenario is
shown in Fig. 4. Usually, there is an integration between
the different pits.
The system is designed to accommodate the arrange-
ment of shovels and loaders, allowing for flexibility in their
location based on operational requirements and the daily
mining plan. However, the crushers have fixed positions
and cannot be easily relocated. Dumps serve as storage
areas for waste or low-grade materials and are typically
situated outside the main processing area, where they are
stacked in piles. Additionally, truck distribution within the
mine is determined by the Dispatch system configuration,
which ensures alignment with the operational plan.
In mining operations, a crucial task is to transport gra-
ded material to the crushers. Currently, there is a sufficient
amount of equipment in place to maintain the objective of
delivering material to the crushers when all equipment is
operational. However, not all trucks are dedicated to
transporting material to the crushers due to the need to
prioritize the removal of waste material in order to access
the ore. This process is carried out on a daily basis. While
Neural Computing and Applications (2024) 36:4523–4535 4527

Fig. 4 Mining operation of two open pits (Image from Google Earth)

there are no difficulties in covering the crusher with trucks

during hours when all equipment is available, specific
periods such as shift changes or lunch breaks pose chal-
lenges. As a result, there is a decrease in truck coverage at Fig. 5 Equipment stoppage process during shift change
the crushers, leading to a shortage of trucks and shovels
available for this specific task.

3.1 Shift change

The mining operations are carried out around the clock,

divided into two 12-hour time periods, with a shift change
occurring at 07:30 and 19:30. To ensure continuous supply
to the crushers, a systematic arrangement is implemented,
positioning all trucks and shovels strategically during these
events. Due to the inability to execute these events
simultaneously, the shift changes are scheduled with a time Fig. 6 Timeline of the shift change process entering operators
gap between them. In Fig. 5, a graph illustrates the distri-
bution of equipment downtime during shift change hours, 3.3 Equipment: trucks, shovels and crushers
showcasing both rounds. To minimize the movement of
personnel, designated areas are designated as parking zones Using the Dispatch system, the operation time of each
during shift change hours, allowing for equipment down- truck is obtained from the mining companies. The trucks
time (see Fig. 6). are classified into categories to calculate the percentage of

3.2 Lunch break

During daily mining operations, there is a scheduled

equipment shutdown for lunch break. Due to the inability
to stop all teams simultaneously, the shutdown is carried
out in three separate rounds. Each round involves approx-
imately one-third of the fleet, with designated time slots
from 11:30 to 12:30, 12:30 to 13:30, and 13:30 to 14:30.
Supervisors are responsible for making decisions regarding
which teams will be taken out of operation in each round.
In Fig. 7, a graph illustrates the minute-by-minute distri-
bution of trucks during the lunch break process, clearly
indicating the specific times when trucks are affected by
the lunch break downtime.

Fig. 7 Snack hours Process—Usual Day

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

4528 Neural Computing and Applications (2024) 36:4523–4535

availability and usage of the equipment. During the cal- in one of the crushers on shift change schedules and lunch
endar time T c , these categories are divided into available breaks, as shown in Fig. 11. In the month evaluated, the
time T D and wasted time (unavailable). The available time event of delays had an impact of 12% on the use of one of
is further broken down into subcategories: effective oper- the crushers. The ‘‘Upstream Delays’’ is the time when the
ating time T O , operational delays, and non-operational crusher did not have a truck, as shown in Fig. 12.
delays. This breakdown allows for the calculation of the
proportion of available time that was effectively utilized by 3.4 Proposed methodology
the equipment, which is termed equipment utilization (see
Eq. 3). This groundbreaking research focused on applying deep
TD reinforcement learning, specifically Q-learning, within the
D% ¼  100 ð3Þ context of trucks, shovels, and crushers in mining opera-
TC
tions. By incorporating information on hauling cycle times,
The percentage of utilization U% is defined as the the agent function was assigned to the trucks, enabling
relationship between the operative time T 0 and the avail- them to make informed decisions. The actions available to
able T D (see Eq. 4) the trucks involved selecting and approaching the available
TO shovels. The research utilized the open-source program-
U% ¼  100 ð4Þ
TD ming language Python [16], along with relevant libraries
that facilitated the simulation of the mining process and
This study was done based on the information from one
training of the deep learning models. This innovative
month of the current year (2022) to calculate the number of
approach holds immense potential for revolutionizing the
trucks operating in the mine per hour, where the operative
mining industry, enhancing productivity, and driving
times T_0 of all trucks were set for each hour, that allowed
transformative advancements.
the estimation of the number of trucks operating in the
The methodology employed in this research encom-
mining company during that hour. Figure 8 shows how the
passed a comprehensive analysis of the hauling cycle over
number of trucks decreased during the shift change and
a 30-day period. Through the utilization of a transact-SQL
lunch break process, regardless of the time of shift.
script and the Dispatch system, essential data on supply
Although the schedules were similar for shift change and
times, shift change delays, truck transfers, shovel loading
lunch break events, day and night shifts had different
and unloading, as well as crushers and dumps were
characteristics as shown.
extracted. The Dispatch system, integrated with GPS glo-
Shovel operators also changed shifts and lunch breaks,
bal positioning sensors, provided accurate information,
and in these cases, the operators usually had some relays so
while 15 shovels with flexible positions, four dumps, three
that the shovels were operating always. However, by
crushers, and a designated truck parking area were strate-
simple inspection, it can be seen from Fig. 9 that the
gically selected. By converting coordinates to the WGS84
number of shovels available at shift changes and lunch
cartographic map and considering 252 combinations for
breaks dropped by up to four units. All the previous pro-
loading and unloading locations, a wide range of daily
cesses determine the tonnage per hour fed to the crushers.
operating scenarios was accounted for. An additional 50
The day and night shifts and the number of trucks unloaded
combinations were used for the truck parking location,
per hour in the crushers can be seen in Fig. 10.
optimizing proximity to the launch area, shovels, and
For the crushers, an event called ‘‘Upstream Delays’’
crushers. This meticulous methodology ensured a com-
indicates the percentage of time in which there were no
prehensive and reliable foundation for subsequent deep
trucks in the crushers. In addition, this event had an impact
reinforcement learning modeling and simulations.

Fig. 8 Operating trucks per day in different shifts Fig. 9 Number of shovels available at shift changes and lunch break

123

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

Neural Computing and Applications (2024) 36:4523–4535
Fig. 10 Number of discharges per day in crushers
Fig. 11 Event upstream delays
Fig. 12 Impact of delays upstream in one of the crushers
The accommodation time of trucks at the shovel refers
to the duration from the truck’s arrival at the shovel until it
is positioned to be loaded. In the various conducted tests,
the average preparation time for multiple trucks was found
to be 1.09 min, with a standard deviation of 0.54 min. To
obtain data on shovel loading times, the information was
divided into two categories. The first category focused on
parking time at the shovels, yielding an average of 1.30
min with a standard deviation of 0.45 min. The second
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
4529
category involved the loading delay time for trucks at the
same shovel, resulting in an average of 4.80 min with a
standard deviation of 1.32 min.
The scheduling of dumps was consistent and unre-
stricted, resulting in queues during material dispatch with a
mean of l = 1.62 min and a standard deviation of r = 0.45
min. Conversely, the crushers experienced truck queues
and had restrictions on equipment access. The discharge
time values for crushers were l = 2.25 min with a r = 0.89
min. This higher value signifies the variability in discharge
times, which are influenced by the material’s mechanical
characteristics, including its resistance and reaction to
force during the crushing process.
The study analyzed the delay times in the mine over a
30-day period in 2022. These delays, including truck cycle
times, were found to be variable and not fixed values. To
account for this variability, a normal distribution curve was
generated using the mean and standard deviation, and these
delays were incorporated into the discrete event simulator.
For the lunch break, the analysis revealed an average
delay of l = 49.77 min with a standard deviation (r) of
6.85 min. During the shift change from 6:30 a.m. to 7:30
a.m. for the day shift, the average delay was l = 25.72 min
with a standard deviation of r = 13.18 min. Similarly,
during the shift change from 6:30 p.m. to 7:30 p.m. for the
night shift, the average delay was l = 25.92 min with a
standard deviation of r = 14.79 min.
3.5 Deep learning model
In Q-learning, the neural network plays a crucial role by
receiving the observations of each agent in the environ-
ment. The network’s output consists of Q values associated
with each possible action that the agent can take. In this
research, the observations for each agent were carefully
defined to capture relevant information. These observations
included travel times to each shovel (15 data points),
queues at each shovel (15 data points), equipment avail-
ability on the route to each shovel (15 data points), queues
at each crusher (three data points), and the number of other
agents present in the environment (one data point). The
outputs of the neural network represented the Q values
directed toward each possible action, based on the agent’s
current observation. Considering that the mine operated in
an unstable environment, the agent had an infinite range of
possible observations. The neural network was responsible
for calculating the Q value based on the current state,
resulting in a network output of 15 points, corresponding to
the number of shovels in the mine.
Significant developments were made in the reward
function (R_t) to accurately capture the performance of the
agents. The reward value was determined based on two
essential variables. Firstly, the total cycle time (DT_cycle)
123
4530
from when the truck was assigned to a shovel until it
unloaded and requested a new assignment. This factor
accounted for the impact of truck queues at the crusher or
dump destinations. Consequently, locations with longer
queues received lower rewards, ensuring an incentive to
minimize delays. The second variable, the type of material
carried by the agent (T_material), was considered. It was
defined as either 0 or 1, representing waste or minerals,
respectively. This factor allowed the reward function to
differentiate between different material types and their
significance for the mining operation. Additionally, an
additional 50% reward was provided when the agent suc-
cessfully loaded ore and chose an ore shovel as an action,
followed by a direct discharge to the crushers. This
encouraged efficient handling of valuable ore material.
Overall, the reward function (R_t) was calculated as per
Eq. 5, incorporating these variables and considerations.
This comprehensive approach ensured that the rewards
aligned with the desired objectives of minimizing cycle
times, optimizing material types, and prioritizing efficient
operations.
T material þ 2
Rt ¼ ð5Þ
DT cycle
A neural network model with 49 inputs, 4900 neurons in
the hidden layer, and 15 in the output layer was created
(see Fig. 13). The training parameters were set as follows:
• Learning rate a ¼ 0:001
• Probability strategy e ¼ 0:8.
• Mean-squared error (MSE) loss function and an Adam
optimizer
The latter helped to update the weights according to the
training data in order to reduce the error during network
learning [17].
The starting conditions of the mine, such as the envi-
ronment, states, actions, shift changes, lunch breaks,
actions, and active equipment, were known. In addition, the
processes carried out during the haul cycle are explained in
detail as follows:
1. The status that trucks own during the haul cycle:
a. When the status was 0, trucks traveled from the
point of origin to the nearest shovel.
Fig. 13 Neural network design—network outputs
123
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
Neural Computing and Applications (2024) 36:4523–4535
b. When the status was 1, the truck was parked next
to the shovels and the materials were being loaded.
c. When the status was 2, the trucks followed their
preset paths to reach the crushers if transporting the
mineral, or they reached the landfill. Then, a status
of three materials was unloaded. Then, the trucks
returned to the point of origin and started a new
cycle.
2. The duration times in the haul cycle used the normal
distributions described in Sect. 3.4, depending on the
status of the trucks.
3. The loss of operational equipment was considered
during lunch break hours (11:30 a.m.–2:30 p.m.) and
shift change hours (6:30 a.m.–7:30 a.m., 6:30 p.m.–
7:30 p.m.).
4. Therefore, some conditions were carried out inside the
mine, which correspond to the input layer of the neural
network model, as detailed below:
a. The travel times of the empty trucks located
around 15 shovels.
b. Number of trucks in each shovel.
c. Number of trucks in each crusher.
d. Assign the paths of travel to each shovel.
5. The shovels were assigned, and the reward was given
when a truck was unloaded (it had no material). It was
established using the greedy epsilon strategy, by
randomly designating a probability of e ¼ 0:2; or
1  e. The network estimated the maximum value Q of
the 15 shovels and stored the actions in a vector.
Finally, the dumps and crushers were assigned.
The following parameters were designated:
• Memory size, equal to the total number of trucks (110
units).
• The number of epochs was set to 2500 (design
criterion).
• Experience vector, where all the states, actions, and
rewards during a simulation day in the mine were
saved.
• The trucks were considered to be the agents. Individual
rewards per agent were defined, and their values were
based on the time from the assignment of the truck to a
shovel to the unloading of the truck, either in the
crusher or dump including the queue that the truck
faced at a particular destination. Therefore, a place that
had a long queue would be lightly rewarded. Next, an
additional 50% reward was provided when the agent
moved to the ore shovels and discharged their content
directly into the crusher.
When the experience size (the accumulated rewards)
after more than 2500 epochs of training exceeded a
Neural Computing and Applications (2024) 36:4523–4535
particular value (10,000), a sample was selected and
entered in mini-batches to generate the Q-type learning
algorithm.
Finally, the loss function was calculated, and the neural
network model was stored in a file.
4 Results
During each training period, the total discharges of the day
were recorded to understand the material movement pat-
terns, as depicted in Figure 14. By adjusting the model’s
parameters, it was observed that the number of discharges
increased. However, there were also instances of low dis-
charge values, particularly at higher training times. This
behavior can be attributed to the implementation of a
greedy epsilon strategy. The greedy epsilon strategy is
commonly employed when training Q-learning models
[18, 19]. It involves balancing the exploration-exploitation
trade-off by considering the value of epsilon during both
training and verification phases. A low value of epsilon
leads to delayed convergence to the optimal solution, while
a high value may result in suboptimal solutions being
favored even when the optimal solution has been found.
Hence, the observed peaks of low discharge values at high
training times can be attributed to the interaction between
the greedy epsilon strategy and the learning process. This
behavior is expected and aligns with previous findings in
the field. It demonstrates the importance of carefully
adjusting the parameters and balancing exploration and
exploitation to achieve optimal performance in the Q-
learning model.
The hauling events in the mine, spanning a full day
equivalent to 86,400 s, were simulated using the Pygame
module. This simulation allowed for the observation and
analysis of the mining operations over a continuous period.
The speed of the operation was controlled, enabling a
Fig. 14 Number of total downloads per day for each training epoch
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
4531
detailed examination of the situation at any given moment.
This approach facilitated a comprehensive understanding
of the dynamics and performance of the mine’s hauling
activities throughout the day. Figure 15 provides a visual
representation of the simulated events, aiding in the visu-
alization and interpretation of the results.
Samples were collected at intervals of 10 s to monitor
the progression of the mining operations. Notably, the total
number of loaded trucks decreased to 55 units during the
shift change event, which occurred at approximately
25,000 and 70,000 s, equivalent to 18:56 and 19:26 h,
respectively. This decline in the number of loaded trucks
can be attributed to the transition between shifts. Similarly,
during the lunch break event, it was estimated that around
75 trucks were present between 40,000 and 50,000 s, cor-
responding to a time range of 11:06–13:53 h. Figure 16
illustrates these fluctuations in the number of trucks during
the lunch break period. By analyzing these samples and
observing the temporal patterns of truck availability,
valuable insights can be gained regarding the impact of
shift changes and lunch breaks on the mining operations.
Such information can aid in optimizing scheduling and
resource allocation strategies to enhance overall efficiency
and productivity in the mine.
The mining operations were simulated for an entire day,
and the analysis focused on identifying periods when the
crushers were without trucks. To determine this, delays
upstream were calculated using the simulation results. It
was found that during all hours of the day, these delays
accounted for less than 10% of the total time, as depicted in
Figure 17. This analysis sheds light on the efficiency of the
mining operations, highlighting that the crushers were
consistently supplied with trucks throughout the day. The
results demonstrate that delays in the upstream processes
Fig. 15 Discrete event simulation of the mine
123
4532
Fig. 16 Simulation of daily operating trucks
were minimal, indicating a well-optimized system that
ensures a steady flow of material to the crushers. By
maintaining delays below 10%, the mining operations can
sustain high productivity and minimize any disruptions that
could hinder the overall efficiency of the process. This
information is crucial for making informed decisions
regarding resource allocation, scheduling, and process
improvements in order to maximize the mine’s
profitability.
The results allowed for estimating the utilization of the
crushers per hour. It was observed that during hours
without any specific events, the crushers operated at close
to 100% capacity. In the cases of shift change and lunch
break events, it was verified that the crushers operated with
over 90% of the trucks, as shown in Figure 18. These
findings demonstrate the effectiveness of the mining
operations in terms of utilizing the crushers efficiently. The
high utilization rates indicate that the crushers are effec-
tively processing the material and contributing to the
overall productivity of the mine. Even during shift changes
and lunch breaks, when some trucks may be temporarily
unavailable, the crushers still maintain a high operating
capacity, ensuring minimal downtime and maximizing
material processing. The results provide valuable insights
into the performance and efficiency of the crusher opera-
tions, enabling informed decision-making regarding
Fig. 17 Percentage of crusher time without truck
123
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
Neural Computing and Applications (2024) 36:4523–4535
Fig. 18 Percentage of use of crushers
resource allocation and operational strategies. By main-
taining high utilization rates, the mine can optimize its
production capabilities and enhance its profitability.
Lastly, tests were conducted to assess the response time
of the algorithm. A total of 1000 executions of the model
were performed, recording the execution time for each
case. The results demonstrated exceptionally fast response
times, with an average of less than 1 ms. The average
response time was calculated to be l = 0.76 ms, with a
standard deviation of r = 2.32 ms, as shown in the test
results. These findings underscore the algorithm’s effi-
ciency and suitability for real-time applications. With
response times measured in milliseconds, the algorithm can
swiftly process and provide optimal solutions to the chal-
lenges encountered in mining operations. The consistently
low execution times ensure that the algorithm can effec-
tively and rapidly handle the complexities of the mining
environment, contributing to enhanced productivity and
profitability. The fast response times exhibited by the
algorithm are a testament to its effectiveness in real-world
mining scenarios, where timely decision-making and
optimized resource allocation are paramount.
5 Discussion
Our research, employing a deep Q-learning reinforcement
learning model in a simulated environment calibrated with
real mining operation data from Peru, stands at the fore-
front of innovations in fleet management optimization. This
approach is particularly relevant when juxtaposed with
similar methodologies in recent studies.
In [7], they proposed a multi-agent contextual rein-
forcement learning framework for large-scale fleet man-
agement in ride-sharing platforms, using a simulator
calibrated with real historical data from Didi Chuxing.
Similarly, [8] presented an actor-critic approach for short-
term production planning and fleet management in mining
complexes, also utilizing a simulator based on historical
data. These methodologies, while analogous in their use of
sophisticated simulation environments, diverge in their
Neural Computing and Applications (2024) 36:4523–4535
specific focus and application areas, highlighting the ver-
satility of simulation-based research in operational
management.
In terms of data utilization, in [7] leveraged historical
data from Didi Chuxing’s travel orders and vehicle tra-
jectories, and [8] relied on historical equipment perfor-
mance data from a large-scale mining operation. Our study,
however, focused on comprehensive haul cycle informa-
tion over a 30-day period from a mining operation in Peru.
This unique dataset allowed for a more detailed and
localized analysis of fleet management challenges, pro-
viding insights specific to the operational contexts of
Peruvian mines.
The results of these studies underscore the efficacy of
simulation and advanced modeling in improving opera-
tional outcomes. Lin et al. [7] achieved significant
enhancements in revenue and customer satisfaction by
coordinating thousands of vehicles. de Carvalho and
Dimitrakopoulos [8] reported a 47% increase in cash flow
by adapting shovel and truck assignments. In comparison,
our research achieved an impressive 90% supply coverage
to crushers during shift changes and lunch breaks,
demonstrating the practical benefits of our Q-learning
model in a real-world mining context.
Regarding precision, the simulation used by [7]
achieved an r2 calibration of 0.9331 between the simulated
and real gross merchandise value. de Carvalho and Dimi-
trakopoulos [8] saw a 5.2% improvement in the forecasts of
fed material quality using data assimilation. Our solution,
with average response times of less than a millisecond per
truck assignment, indicates high precision and suitability
for real-time applications. This aspect of our research is
particularly noteworthy, as it suggests the potential for
immediate practical implementation in the mining industry.
Comparatively, [20] presented a multistage optimization
framework for truck assignment in open-pit mines, using
goal programming and simulation, while [9] developed a
two-stage model for real-time truck dispatch, combining
flow network theory and integer programming. While these
studies offer valuable insights into the complexity of fleet
management, our research’s use of deep Q-learning in a
simulated environment to optimize crusher supply cover-
age demonstrates a novel approach that effectively bal-
ances theoretical robustness with practical applicability.
Furthermore, our solution’s capacity to maintain over
90% supply coverage to crushers during crucial periods,
such as shift changes and lunch breaks, represents a sig-
nificant advancement over [20], who reduced upstream
delays in crushers to less than 10% throughout the day. The
average response time of less than a millisecond per truck
assignment in our study not only underscores the precision
of our model but also its potential for real-time application
in the mining industry.
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
4533
In summary, our study contributes to the growing body
of knowledge in mining fleet management by introducing a
novel, efficient, and highly precise method. By integrating
advanced machine learning techniques and leveraging
comprehensive data, we have demonstrated a significant
improvement in operational efficiency, providing a
promising direction for future research and practical
applications in the field.
6 Conclusions
This study has yielded critical insights into the optimiza-
tion challenges within mining operations. We observed
notable reductions in operational efficiency during shift
changes and lunch breaks, with a decrease of approxi-
mately 45–60% and 35–45% in the number of operating
trucks, respectively. Concurrently, a reduction of one to
four units per day was noted in shovel availability. The
study also identified additional delays, including planned
and operational losses, particularly affecting crushers and
equipment.
To address these inefficiencies, we designed a Q-learn-
ing deep reinforcement learning model, treating each truck
as an independent agent with actions corresponding to the
available number of shovels. The model’s framework
included key operational parameters such as travel times to
shovels, truck queues at shovels and crushers, and the
distribution of trucks en route to each shovel. A reward
system was implemented, prioritizing ore transportation
efficiency and minimizing cycle times, thereby fostering
lower truck queues and more efficient material handling.
Our simulation environment was meticulously con-
structed to mirror real-world conditions, incorporating
actual time distributions for each transport cycle stage and
integrating delays associated with shift changes, lunch
breaks, and travel. The neural network model featured a
complex architecture with 49 input neurons, 4900 neurons
in the hidden layer, and 15 output neurons. The deep
reinforcement learning algorithm trained the network to
estimate the maximum Q-values based on agent
observations.
The results showcased the model’s effectiveness,
achieving over 90% coverage for crusher usage during shift
changes and lunch breaks, while maintaining manageable
truck queues at strategic locations. Furthermore, the mod-
el’s average execution time of one millisecond per truck
assignment to shovels underscores its suitability for real-
time operational applications.
This research marks a significant advancement in min-
ing fleet management through the application of a deep Q-
learning reinforcement learning model. The model’s ability
to enhance productivity, reduce downtime, and increase
123
4534
profitability in the mining industry is a testament to its
potential.
Nevertheless, it is crucial to acknowledge the study’s
limitations and outline future research directions. The
simulated environment, calibrated with real operational
data, does not fully capture the complexities of an actual
mining operation. This limitation underscores the necessity
for additional testing in physical environments to validate
the model’s real-world applicability.
Moreover, the model was tested in a singular mining
operation in Peru. To ascertain its broader applicability and
generalizability, it is imperative to expand the scope of
testing to include diverse mining sites with varying oper-
ational characteristics.
The model’s current limitation, restricting truck actions
to the number of available shovels, presents an opportunity
for enhancement. Exploring more dynamic agent actions,
such as alternative routing in response to fluctuating delays,
could potentially elevate the model’s performance.
In this study, the achievement of rapid response times
demonstrates the model’s potential for enhancing effi-
ciency in mining operations. However, the challenge lies in
replicating this efficiency on a larger scale, particularly in
real-time implementations. Future research endeavors will
be directed toward validating and refining this model under
more diverse and realistic conditions. Testing the model in
actual physical mining environments will provide critical
insights into its practical applicability and effectiveness.
Simultaneously, extending its application across multiple
mining sites, each with its unique operational characteris-
tics, will allow for a comprehensive evaluation of the
model’s adaptability and generalizability. This expansion is
crucial for ascertaining the model’s robustness in varying
mining contexts.
Furthermore, the exploration of more dynamic actions
for truck agents, such as adjusting routes in response to
changing operational conditions, will contribute to
enhancing the model’s responsiveness and overall effi-
ciency. The development of computational optimizations is
also vital for the model’s efficient deployment in large-
scale, real-time applications. This includes incorporating
additional uncertainties, like variations in cycle times and
equipment availability, to augment the model’s realism and
practical utility. An expanded focus on coordinating the
model across multi-pit mining operations will broaden its
scope and enhance its effectiveness in complex mining
scenarios. Lastly, integrating the model with established
commercial fleet management systems, such as MOD-
ULAR MINING, will facilitate further testing and valida-
tion, ensuring its readiness for industry-wide adoption and
implementation. Through these focused efforts, the study
aims to significantly contribute to the advancement of
123
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
Neural Computing and Applications (2024) 36:4523–4535
mining operations, driving productivity and operational
efficiency.
Data availability The data used to support the findings of this study
are available from the corresponding author upon request.
Declarations
Conflict of interest The authors declare that they have no known
competing financial interests or personal relationships that could have
influenced the work reported in this study. This study did not receive
any financial support.
References
1. Moradi Afrapoli A, Tabesh M, Askari-Nasab H (2019) AA
multiple objective transportation problem approach to dynamic
truck dispatching in surface mines. Eur J Oper Res 276:331–342.
https://doi.org/10.1016/j.ejor.2019.01.008
2. de Carvalho JP, Dimitrakopoulos R (2021) Integrating production
planning with truck-dispatching decisions through reinforcement
learning while managing uncertainty. Minerals. https://doi.org/
10.3390/min11060587
3. Mohtasham M, Mirzaei-Nasirabad H, Askari-Nasab H, Alizadeh
B (2022) Multi-stage optimization framework for the real-time
truck decision problem in open-pit mines: a case study on Sungun
copper mine. Int J Min Reclam Environ 36:461–491. https://doi.
org/10.1080/17480930.2022.2067709
4. Bnouachir H, Chergui M, Machkour N et al (2020) Intelligent
fleet management system for open pit mine. Int J Adv Comput Sci
Appl (IJACSA). https://doi.org/10.14569/IJACSA.2020.0110543
5. Baek J, Choi Y (2020) Deep neural network for predicting ore
production by truck-haulage systems in open-pit mines. Appl Sci
10:1657. https://doi.org/10.3390/app10051657
6. Zhang C, Odonkor P, Zheng S, et al (2020) Dynamic dispatching
for large-scale heterogeneous fleet via multi-agent deep rein-
forcement learning. In: 2020 IEEE international conference on
big data (big data). IEEE, Atlanta, GA, USA, pp 1436–1441
7. Lin K, Zhao R, Xu Z, Zhou J (2018) Efficient large-scale fleet
management via multi-agent deep reinforcement learning. In:
Proceedings of the 24th ACM SIGKDD international conference
on knowledge discovery & data mining. Association for Com-
puting Machinery, New York, NY, USA, pp 1774–1783
8. de Carvalho JP, Dimitrakopoulos R (2023) Integrating short-term
stochastic production planning updating with mining fleet man-
agement in industrial mining complexes: an actor-critic rein-
forcement learning approach. Appl Intell. https://doi.org/10.1007/
s10489-023-04774-3
9. Ahangaran DK, Yasrebi AB, Wetherelt A, Foster P (2012)
AReal—time dispatching modelling for trucks with different
capacities in open pit mines. Arch Min Sci 57:39–52. https://doi.
org/10.2478/v10267-012-0003-8
10. Mu C-Y, Chou T-Y, Hoang TV et al (2021) ADevelopment of
multilayer-based map matching to enhance performance in large
truck fleet dispatching. ISPRS Int J Geo-Inf. https://doi.org/10.
3390/ijgi10020079
11. Chaowasakoo P, Seppälä H, Koivo H, Zhou Q (2017) ADigital-
ization of mine operations: scenarios to benefit in real-time truck
dispatching. Int J Min Sci Technol 27:229–236. https://doi.org/
10.1016/j.ijmst.2017.01.007
Neural Computing and Applications (2024) 36:4523–4535
12. Silva L, Torquato M, Fernandes M (2018) Parallel implementa-
tion of reinforcement learning Q-learning technique for FPGA.
IEEE Access. https://doi.org/10.1109/ACCESS.2018.2885950
13. Chen J, Li K, Li K et al (2021) Dynamic bicycle dispatching of
dockless public bicycle-sharing systems using multi-objective
reinforcement learning. ACM Trans Cyber-Phys Syst. https://doi.
org/10.1145/3447623
14. Adi TN, Bae H, Iskandar YA (2021) Interterminal truck routing
optimization using cooperative multiagent deep reinforcement
learning. Processes. https://doi.org/10.3390/pr9101728
15. Modular Mining Systems, Inc (2016) Sistema Dispatch
16. Führer C, Solem JE, Verdier O (2021) Scientific computing with
Python: high-performance scientific computing with NumPy,
SciPy, and pandas, 2nd edn. Packt Publishing Ltd, Birmingham
Mumbai
17. Wilson R, Mercier PHJ, Navarra A (2022) Integrated artificial
neural network and discrete event simulation framework for
regional development of refractory gold systems. Mining
2:123–154. https://doi.org/10.3390/mining2010008
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
4535
18. Bulut V (2022) Optimal path planning method based on epsilon-
greedy Q-learning algorithm. J Braz Soc Mech Sci Eng 44:106.
https://doi.org/10.1007/s40430-022-03399-w
19. Zai A, Brown B (2020) Deep Reinforcement Learning in Action,
1st ed. Manning, Shelter Island, New York
20. Moradi Afrapoli A, Askari-Nasab H (2019) Mining fleet man-
agement systems: a review of models and algorithms. Int J Min
Reclam Environ 33:42–60. https://doi.org/10.1080/17480930.
2017.1336607
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
Springer Nature or its licensor (e.g. a society or other partner) holds
exclusive rights to this article under a publishing agreement with the
author(s) or other rightsholder(s); author self-archiving of the
accepted manuscript version of this article is solely governed by the
terms of such publishing agreement and applicable law.
123
Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:

1. use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
2. use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
3. falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
4. use bots or other automated methods to access the content or redirect messages
5. override any security feature or exclusionary protocol; or
6. share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at

onlineservice@springernature.com