Optimising Public Bus Transit Networks Using Deep Reinforcement
Learning
Ahmed Darwish1 , Momen Khalil1 , and Karim Badawi2
Abstract— Public Transportation Buses are an integral part
of our cities, which relies heavily on optimal planning of routes.
The quality of the routes directly influences the quality of
service provided to passengers, in terms of coverage, directness,
and in-vehicle travel time. In addition, it affects the profitability
of the transportation system, since the network structure
directly influences the operational costs. We propose a system
which automates the planning of bus networks based on given
demand. The system implements a paradigm, Deep Reinforce-
ment Learning, which has not been used in past literature
before for solving the well-documented multi-objective Transit
Network Design and Frequency Setting Problem (TNDFSP).
The problem involves finding a set of routes in an urban area,
each with its own bus frequency. It is considered an NP-Hard
combinatorial problem with a massive search space. Compared
to state-of-the-art paradigms, our system produced very com-
petitive results, outperforming state-of-the-art solutions.
Index Terms—Deep Reinforcement Learning, Attention Mod-
els, Transit Network Design, Frequency Setting
I. INTRODUCTION
ublic transportation is an important component of any
P urban city, and its quality naturally positively affects
the well-being of the city’s residents. However, enhancing
the quality of a public transportation system in terms of
coverage and frequency comes at a cost to the operator, who
needs to maintain more vehicles with more employees to
operate them, which maps to extra Capital and Operating
Expenditure (CAPEX and OPEX). The presence of conflict-
ing interests makes the process of planning a public transit
network challenging.
Planning a bus-based transportation network is even less
straightforward, since the infrastructure is less rigid than,
for example, a rail-based mode of transportation. And it
can even be more complicated if the infrastructure is still
undetermined, i.e., the locations of the bus stops are still
unknown. This, however, is outside the scope of this piece of
literature, and we are mainly concerned with the optimisation
of networks with determined infrastructure. According to [1],
the design phase of such networks can be divided into five
main stages: network design, frequencies setting, timetable
1 Ahmed Darwish (https://orcid.org/0000-0001-5793-4128) and Momen
Khalil (https://orcid.org/0000-0001-7837-1870) are members of the R&D
department of Idea in Motion Inc. They are also studying at the Computer
Science and Engineering Faculty at the German University in Cairo (GUC),
Egypt. ahmed.ahmeddarwish@student.guc.edu.eg
momen.khalil@student.guc.edu.eg
2 Dr. Karim Badawi is currently with Trapeze Group, Switzerland.
He is also teaching and heading a research group at the department
of Information Technology and Electrical Engineering at the
Swiss Federal Institute of Technology Zurich (ETH Zürich)
in Switzerland. karim.badawi@trapezegroup.com
badawi@iis.ee.ethz.ch
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.
development, bus scheduling and driver scheduling. The
stages are highly dependable on each other, and the output
of the last stage can be the starting point of another iteration
of the planning process. Each stage can be thought of as
a combinatorial problem, with a computational complexity
lying in the NP-Hard domain[2]. This makes it almost
impossible to optimise the whole pipeline with our current
computational resources using one single algorithm.
Throughout the years, different algorithms have been
proposed to optimise one or more stages separately from
others, using a whole spectrum of different approaches.
However, they relied heavily on hand-crafted heuristics.
Naturally, these heuristics would come from the intuition
and the experience of an expert. This can be a limitation,
since one’s perception of reality may not always be the most
accurate. One common sub-problem is concerned with the
optimisation the first two stages together, what is known
in literature as the Transit Network Design and Frequency
Setting Problem (TNDFSP). This problem aims to come up
with the most efficient network topology, along with the
respective frequencies of each line, which is based on the
customers’ travel demands.
In this paper, we propose a novel approach to solve the
TNDFSP using Deep Reinforcement Learning (DRL), by
exploiting the inherent combinatorial nature of the prob-
lem. Our approach utilises a modified Multi-Head Attention
(MHA) network to output a transit network, which is post-
processed and input to the reward function that outputs the
scalar values needed for back-propagation. Our model does
not require any heuristics to be included in the algorithm,
and it also allows us to prioritise the objectives to optimise
(either customer satisfaction or operator cost minimisation),
by fine-tuning a few custom hyperparameters before training.
Our main contribution in this paper is the use of DRL
for solving TNDFSP to output a complete, optimised net-
work with frequencies, with no need whatsoever for pre-
determined heuristics. We trained and evaluated our model
on Mandl’s benchmark network[4].
II. LITERATURE REVIEW
A. Different Approaches for solving TNDFSP
Purely mathematical models have been designed and built
to design the network routes. However, they keep being too
strict for the dynamic always-changing nature of the real
world, and they generally only work most efficiently for the
targeted variant of the problem[3].
Another group of solutions utilised hand-crafted heuristics
to optimise the network. For example, such heuristics would
be used in determining the topology of the initial population
for a Genetic Algorithm[6].
One of the earliest significant efforts was carried out by
Mandl[4], who applied his heuristics to a real network in
Switzerland. This network is considered a benchmark for
evaluating algorithms up until now. Baaj and Mahmassani[5]
proposed more complex heuristics than those of Mandl’s,
but they still remain too sophisticated and require a good
understanding of the transportation system being optimised,
and are not generally integrable with a lot of real-world
networks.
Evolutionary Algorithms have also shown significant ef-
fectiveness in solving this problem, abstracting away much
of the needed knowledge, and having more applicability
to real-life. Chakroborty and Diwedi[6] proposed a Genetic
Algorithm (GA) with a multi-objective fitness function, fo-
cusing on user costs and coverage, which outperformed its
predecessors, but only solved the routing problem.
Many of the algorithms which followed started focus-
ing more on the TNDFSP. Zhao and Zeng[8] combined
Simulated Annealing, a probabilistic technique, with a GA
to acquire improved results. Kidwai et al.[7] proposed an
iterative multi-stage algorithm which minimises head-ways
and the routes. Arbex and da Cunha also proposed a similar
algorithm, with the operator cost as a part of the constraints,
in order to minimise the fleet size[9]. Jha et al.[10] applied
multi-objective particle swarm optimisation with multiple
search strategies algorithm[11] in designing Transit Net-
works, with an additional objective of minimising the CO2
emissions of the buses, and produced state-of-the-art results
for relatively larger number of lines when compared to the
results in the paper by Arbex and de Cunha, but still remains
inferior when the number of routes is 4.
B. Solving combinatorial problems using DRL
All the papers we surveyed involved pre-determined
heuristics which are used to traverse the search tree of
solutions until a good enough solution was found. Based
on some insights from the survey by Bengio et al.[12], we
decided to experiment with the possibility of learning those
heuristics using connectionist approaches. The survey did not
discourage the use of this approach for highly-dimensional
problems like the TNDFSP.
Several papers have been found trying to solve different
graph combinatorial problems, such as the Travelling Sales-
man Problem (TSP), the Vehicle Routing Problem (VRP),
and the Orienteering Problem (OP), all of which are, more
or less, very specific instances of our problem.
Bello et al.[13] have utilised a Pointer Network with
an attention layer as an agent, deciding which node to
visit in the TSP problem. The Actor-Critic approach was
used to train the agent. Dai et al.[14] implemented a self-
feeding network which takes as an input the embedding of
a whole graph, while masking out the nodes the agent is
not allowed to pick, and keeps inputting the output back
into the network until the solution is complete. Nazari et
al.[15] proposed removing the LSTM encoder of pointer
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.
network from Bello’s work, since they claim there is no
temporal value in the order the nodes are input to the
network. In addition, they also propose inputting a ”dynamic
element” to the attention mechanism, such that the network
is more robust against dynamic properties of the Vehicle
Routing Problem (VRP), and they utilise the A3C training
method. Kool et al.[16] used the Multi-Head Attention[17]
model instead of the typical single-head attention model
and a different non-connectionist baseline approximator, to
reduce the variance between the loss gradients. They report
state-of-the-art results when compared to other RL-based
Combinatorial Optimisation frameworks.
Kool et al. also mentioned the possibility of using their
neural network architecture with similar problems given
a custom masking and reward function. Consequently, we
decided to employ their architecture for the TNDFSP, due to
its leading performance, and the feasibility of adapting our
problem to the model architecture.
III. METHODOLOGY
A. Problem Statement
As mentioned before, we have two conflicting objectives to
optimise. The first one of them, Customer (or user) Satisfac-
tion, is concerned with a short trip time, a minimum number
of transfers, and a short waiting time at stations. The second
one, Operator costs, is concerned with minimising CAPEX
and OPEX. For simplicity, we are going to assume that the
number of vehicles needed to serve a specific network of
stations is sufficient to represent such costs sufficiently, as
shown in literature. We can formulate the problem in the
following manner:-
Given the tuple I = (G, D, T) consisting of a directed
graph, G = (S, R), where S denotes the set of bus stops, and
R denotes the set of roads connecting them, an OD-matrix
D ∈ R|S| , and a travel time matrix1 T ∈ R|S| , produce a set
of routes, L, where each route is a path in graph G, which
acts as a one of the Pareto optimal solutions minimising the
values U and O, which denote the customer cost, and the
operator cost, respectively.
The way the values U and O are computed will be
explained briefly in the Reward Function section.
The sub-problems of our target problem can not be solved
simultaneously. Instead, they must be solved sequentially,
since Frequency Settings requires the assignment of the
demand of the transportation network to the different routes.
This can not occur if no transit network exists. Therefore,
a network has to be designed first, before calculating the
headway.
Assumptions and Hyperparameters:
To help simplify the search space, we impose additional
non-functional constraints (some in the form of hyperparam-
eters) and assumptions about the nature of our inputs and
outputs. The constraints are similar to those found in Arbex
and da Cunha’s paper[9]:-
1 This matrix contains the time needed to go from a station to another.
 1) The output network is a connected graph. This means
that all stops are reachable from any other stop.
2) Each and every stop in the graph must be present in
at least one of the routes.
3) Routes must be ultimately different from each other.
Identical routes are considered to be a single route with
their frequencies combined.
4) No route can have a cycle.
5) The number of output routes, |L|, is predefined by the
operator.
6) All buses in the operator’s fleet have the same capacity.
7) The number of stops in each route has to be within
a predefined range. In addition, the estimated travel
time to traverse a single route has to be bounded by a
predefined value. All this is to account for feasibility of
bus schedules and to avoid overtiring the bus drivers.
B. Model Architecture Overview
The neural network architecture we are using in this paper,
which is proposed originally by Kool el al., is based on
the Encoder-Decoder Architecture, in addition to Attention
Mechanisms (AMs).
Initially, an encoder similar to the one in the Transformer
architecture proposed by Vaswani et al.[17] is used to pro-
duce the nodes and the graph embeddings, but without the
use of the positional encoding, since it was unnecessary to
our purpose. The initial embeddings are then put through N
attention layers, each consisting of two sub-layers: Multi-
Head Attention and a fully connected Feed-Forward layer to
perform edge-wise projections. Following that, the encoder
calculates a graph embedding as the mean of the final node
embeddings. Then, all of the embeddings are used as input
to the decoder.
The decoder depends on the embeddings of the encoder,
the context vector, as well as the previously output values for
deciding on the current decoding step. Decoding happens
sequentially, where the context node embedding is passed
from the encoder at each time step, along with the graph
embeddings.
In each decoding step, the attention values are then com-
puted and used to calculate the log-probabilities:-

−∞
 T  for disallowed input
u(c)j = q(c) kj (1)
C. tanh √ d
otherwise
k
C is a constant used for clipping the values before masking.
dk is the query/key dimensionality, which is equal to the
dimensions of the embedding layer output, divided by the
number of attention heads.
Finally, the normalised probability vector, p, is computed
using softmax, after which the output is determined using
either sampling or greedy approaches:-
pi = sof tmax(u(c)i )
After the calculation of the reward of the output for some
instance s, gradient descent is employed to minimise the loss,
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.
L(θ), where θ represents the neural network’s weights. The
REINFORCE approach is used to approximate the gradient,
with a baseline, b(s). Moreover, Kool et al. proposed a rollout
baseline with periodically changing baseline policy. Thus, for
a probability distribution, pθ (π|s), using policy π to solve an
instance s, we have:-
∇L (θ|s) = Epθ (π|s) [(L (π) − b(s)) ∇ log pθ (π|s)] (3)
C. TNDFSP-specific Model Configurations
In this section, we will explain how an instance of our
problem is transformed to a combinatorial optimisation prob-
lem that can be solved by the model.
1) Encoder Input: To provide a better context to the
decoder, the input graph is augmented with the demand
and travel time information, which are better represented as
edges, since the nodes have no useful intrinsic properties.
In such case, the demand and the travel matrices can be
thought of as adjacency matrices representing edges of the
road network. The nodes of the graph are, consequently,
redundant, and will not be input to the model, but will
be used for the mask function, and calculating the reward.
The model, as a result of this change, will be outputting a
sequence of consecutive edges from the graph.
Two more inputs are the OD-matrix, δ, and a modified
travel time matrix, d. The original travel time matrix corre-
sponds to the travel time values between the nodes in the
graph that already have a link between them. However, in
order for the model to capture all the information about the
travel times in the graph, a new travel time matrix needs
to be created to include all the travel times between all
the nodes even if no direct link exist between them. This
task is achieved by applying Floyd-Warshall’s shortest path
algorithm[18][19] to the graph. The original matrix would
later be used in the masking function.
To be able to differentiate between the different routes,
and have the model learn how to optimise separate routes.
A special “delimiter” edge is used and appended to the
beginning of the list of edges, which the model is forced to
output before starting a new route. This edge’s embeddings
are calculated separately from the rest of the edges, using
separate parameters, due to its special purpose.
The delimiter edge is input before the other edges, giving
it an index of zero in the array of edges, x. Thus, the initial
embeddings of our input is calculated through the learned
linear projection of parameters through parameters W x and
bx in the first embedding layer as follows:-
(
(0) W0x xi + bx0 if i = 0
hi = (4)
W x [xi , δi , di ] + bx if i = 1, ..., n(n−1)
2
(2)
[.] represents horizontal concatenation, and indexing the
matrices using i is equivalent to accessing the value of the
cell corresponding to edge i. n is the number of graph nodes.
 2) Decoder Context: For our problem, additional values
are added to the context vector, most of which were men-
tioned in the Assumptions and Hyperparameters section.
The last decoded edge is added to the context vector. For
the first decoding step, that would be the delimiter edge,
since we assume that a route would be starting there. This
piece of information and the other values are concatenated
to the embeddings in the following fashion:-
h i the pre-defined transfer penalty of the network. The edges’
 h̄(N ) , h(N ) , r, nmin , nmax , len, U
(N ) 0
h(c) = h i transfers” attribute, which is 1 only for the transfer edges.
 h̄(N ) , h(N )
πt−1 , r, nmin , nmax , len, U
where:
• nmin and nmax represent the remaining number of edges
left to reach the predefined minimum and maximum
number of edges per route, respectively.
• len is the travel time needed to traverse the route being
decoded up till the last decoding step.
• r represents the number of routes remaining needed to
reach the pre-defined value.
• U is the number of unique nodes in all decoded routes
so far. This is needed to be able to satisfy the first
assumption.
The values nmin , nmax , and len, are reset every time the
delimiter edge is decoded, as to start a new route.
3) The Masking Function: The main purpose of the
masking function is to make sure the constraints mentioned
in Assumptions and Hyperparameters are met to make
sure that the decoded output is valid. In addition, we need
to ensure that the chosen edges are consecutive, and actually
exist in the original road network. This is done by keeping
track of all visited nodes and edges, as well as a count of
finished routes, and number of nodes accumulated so far in
the current route.
4) Output Pre-processing and Reward Calculation: After
the decoding process is finished and the solutions are formed,
the costs of the decoded solutions, which denote the scalar
values U and O in the problem formulation, must be
calculated.
However, these calculations are not straightforward, and
require a lot of additional information which are inferred
from solutions. Information about travel times, or rather
distances, between the different pairs of stops in the solution
network is required. In addition, knowledge of the exact paths
taken to achieve a trip between any two pair needs to be
gathered. This knowledge constitutes the number of transfers
needed in a trip and the lines used. To be able to calculate the
reward value of the decoded solution, we therefore require
each of the following:-
1) A travel-time matrix for the decoded network, N T T ,
which have the time needed to transfer from one line
to another included in the values.
2) A matrix representing the needed number of transfers
for each OD pair, T P P .
3) A version of the output graph where the demand is
assigned to the edges, DAG. In other words, the
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.
amount of demand passing through each edge in the
graph is calculated, as to be able to calculate the peak
load of each of the lines, to be able to find their
respective headways.
For an accurate computation of requirements 1 and 2, we
expand the graph, making copies of any node which lies
on different lines, and adding extra transfer edges between
them. The travel time value of those transfer edges would be
if t = 1 weights are also modified, having an extra “Number of
if t > 1 An example of this operation, which we denote as Graph
(5) Extender, is shown in Fig. 1.
Once the graph is modified, we run a modified version
of Dijkstra’s Algorithm[20], which does not only output
the shortest paths and their lengths, but also the number
of needed transfers and the lines IDs corresponding to this
shortest path. The output IDs of the used lines is required
later in the reward calculation.
The algorithm is run for each node in the modified graph,
and the output of each run is accumulated in two (n × n)
square matrices where n is the original number of stops, one
for the travel time values and the other for the number of
transfers. In cases where the algorithm is run on one of the
copied nodes, we always take the best performing copy, in
terms of travel time.
To fulfil the third aforementioned requirement, an exact
copy of the modified graph is created, where all weights
of the edges are initialised with zero. Afterwards, for each
edge in the output paths, the demand value of the given
pair, is added to the weights. Sometimes, it would be equally
efficient to go from point A to point B using the same set of
edges, but through different lines, in this case, we split the
demand among the parallel lines, assuming that they would
be uniformly used by the passengers.
5) Reward Calculation: The following equation is used
to calculate the reward value:
R(π) = W1 · AT T (π) + W2 · P SD(π)+
(6)
W3 · P U D(π) + W4 · N F S(π)
All Wi values represent the priorities of each of the metrics
in the reward function. A higher weight increases the priority
of this specific metric. If all weights are equal, all metrics
are given the same priority. Every addend is normalised
Fig. 1. Modification of an example graph. Each colour represents a line,
while blue represents the transfer edges. As explained, the weight tuple
changes from (travel time) to (travel time, number of transfers). The transfer
penalty for this example is 5.
before being multiplied by the coefficient, in order to reduce IV. EXPERIMENTS AND RESULTS
variance during training. A. Data and Hardware Description
The first addend represents the Average Travel Time for
Our system was implemented using PyTorch2 . We ran our
all satisfied passengers, incorporated with the pre-defined
code on Google Colaboratory3 , which provides users with
transfer penalty, P .
an Intel Xeon processor, a Tesla T4 GPU with a memory of
T IV T (π) + P ×
P
T P P (π) 16GB, a RAM of ∼12GB.
AT T (π) = P (7) Mandl’s Network[4], along with its demand matrix, was
OD
used to train our model. As mentioned in the introduction
where: section, this network is used as a benchmark for papers
X proposing transit network planning algorithms. The network
T IV T (π) = OD ◦ N T T constitutes 15 different Swiss cities, connected by a road
network. The network consists of 21 edges, each having the
The second addend, Percentage of Satisfied Demand, is travel time as a weight. The network is shown in figure 2.
a measure of the percentage of the satisfied trips. It is The passenger demand is represented as a matrix for the
calculated by calculating of trips completed with 0, 1, and 2 peak hour, summing up to a total of 15,570 passengers. For
transfers. To always maximise direct trips, indirect trips are simplicity, the OD matrix is symmetrical and the travel time
added to the measure as a penalty. This function contains for each direction of an edge is equal.
weighting coefficients, ωi , similar to the ones in equation 6.
B. Model Training
P D(π) = ω1 × [−P (T P P0 (π))] + Regarding the model, we utilise the same hyperparameters
(8)
ω2 × P (T P P1 (π)) + ω3 × P (T P P2 (π)) as those used by Kool et al.[16] for learning the Travelling
Salesman Problem:-
where:
• Weights initialised uniformly between
P h √ √ i
OD ∗ T P Px −1/ d, 1/ d , where d is the input dimension.
P (T P Px ) = P
OD • 3 encoder layers, with input and hidden sizes of 128
• 8 attention heads
The function, T P Px , converts a matrix into a Boolean −4
• Learning rate of η = 10
mask where any element equivalent to any of the elements
• The optimiser used is Adam[21] along with Batch
in the list, x, is set to one, otherwise the element is set to 0.
Normalisation
The third addend is simply the Percentage of Unsatisfied
Demand, P (T P Px>2 ). As to TNDFSP-specific hyperparameters, we set the fol-
The fourth and the last addend is the Needed Fleet Size lowing values, which are similar to those found in literature,
for our network, which is dependent on, and can be inferred to make comparisons more valid:-
from, the chosen waiting times per line. The waiting time 2 https://pytorch.org/
is assumed to be half the headway value, which is the 3 https://colab.research.google.com/

reciprocal of the frequency of buses in a given time period.

This frequency can be calculated for a given line using this
equation proposed by Ceder[1]:-

LOADlmax
fl = (9)
LF . CAP
LOADlmax is the maximum load of a given line, i.e., it is
the highest amount of passengers a line has to satisfy in a
given period of time. LF is the pre-defined load factor of
the vehicles’ seats. CAP represents pre-defined the capacity
of the operator’s vehicles.
The maximum load can be easily determined by examining
the graph, DAG, where the edge with the highest demand in
each line is identified. Once we have each line’s frequency,
we can sum all the values to determine the value N F S and
injecting it into the equation.
For trips which involve one or two transfers, we assume
that transfers divides the trips into separate trips, each with
its own waiting time, according to the IDs of used lines
computed by Dijkstra’s algorithm. We then sum up the values
at each segment. Fig. 2. Mandl’s Benchmark Network. Edge weights represent travel times.

Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.
 • Number of routes in the network = {4, 5, 6}
• Minimum number of nodes per route = 3
• Maximum number of nodes per route = 11
• Maximum route length in terms of travel time units =
35
• Transfer Penalty = 25
• Maximum load factor of vehicles = 1.25
• Vehicle Capacity = 40
• Coefficients of the reward function (it should be noted
that these weights would prioritise directness and oper-
ator costs):-
W1 = 2 W2 = 1.5 W3 = 5 W4 = 2
ω1 = 2 ω2 = 1.5 ω3 = 1
For the first experiment, the model was trained on Mandl’s
Network for 100 epochs, each consisting of 30 batches,
containing 256 input instances each. The number of routes
was 4. The batch size could not be made bigger due to
memory constraints, but increasing the size would have
positively contributed to the efficiency of the model training.
On average, a batch took 25 seconds for 4-route training, and
90 seconds for 6-route training. The progress of training the
4-route model is shown in figure 3. As for evaluation, we
decoded 100 instances using a sampling decoding strategy,
and picked the best network.
After training, we experimented using the same model to
output a network of 5 and 6 routes, to get a feel of how well
generalised our model is. However, the model kept outputting
copies of the original four routes.
Following the experiments of the 4-route-model, we de-
cided to continue its training, setting the number of routes
to 6. After the training, we tried outputting networks with a
fewer number of routes. The results were more promising,
and will be be discussed in the following subsection.
C. Results and Comparison to State-of-the-Art Networks
As shown in table I in the next page, the 4-route model
was able to produce very close results to those of Arbex
Fig. 3. Change of the loss value during training the model for 4 routes
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.
and Cunha. When outputting 4 routes, the model was able
to save up operator costs by 14.3%, at an increase of only
11.6% to the average headway, and 0.87% in travel time. In
a practical sense, the increase of the average headway is only
25.8 seconds, which can seem trivial when compared to the
12 buses the operator did not have to use to meet almost the
same directness and travel time. The 6-route model produced
almost the same numbers.
As to networks of 5 routes, the 6-route model was able
to produce a network of superior directness, and requires 5
less buses. Other metrics were insignificantly longer, with
the average travel time being 9.6 seconds longer, and the
average headway being more than a minute longer. Again,
practically, this is negligible in contrast to the reduction in
operator expenditure and the directness.
Finally, our model generated 6-route networks of better
directness with the 6-route model. It also reduced average
travel time by about 35 seconds, and the average headway
by about 20 seconds. On the other hand, the model cost the
operator 4 more buses than the state-of-the-art network.
V. CHALLENGES
One of the most important factors that would make our
deep learning approach more sustainable and quicker to train
on different architectures and reward functions is the concur-
rency of the reward function. However, due to the difficulty
making specific parts of our reward function parallel, like
Dijkstra’s function and the Graph Modifier, training was
slowed down by those sequential bottlenecks. This prevented
us from trying out enough variations of the weights, or trying
to have the model train on outputting a diverse number of
routes. What also necessitates the parallelism of our reward
function is the need for optimising bigger networks, that have
a number of stations in the order of the hundreds.
VI. CONCLUSION
In this paper, we proposed using Deep Reinforcement
Learning to solve one of the oldest well-established problems
in public transportation, the TNDFSP. This approach was
never used before in literature, and it obviates the need for
heuristics of any kind for reaching a good solution. The
proposed system showed great potential of finding optimised
solutions that satisfy both the passengers and operators
objectives. The networks produced by our model were very
competitive in comparison with state-of-the-art algorithms,
when applied to Mandl’s benchmark network.
In the future, we can improve the process of training the
model by finding parallel alternatives to Dijkstra’s and the
Graph Extender, to alleviate the sequential bottleneck. This
will deem the training of the model to generate larger net-
works more feasible. In addition, several Origin-Destination
matrices can be fed to the model during training to allow it
to generalise better to network demand patterns over longer
periods. Consequently, the model will be able to produce
high-quality networks given unseen demand matrices.
 TABLE I
C OMPARISON BETWEEN OUR PROPOSED ARCHITECTURE (DRL) AND THE STATE - OF - THE - ART

Average
Maximum Average
# of # of Direct 1-transfers 2-transfers In-vehicle
Source Headway Headway
Lines Buses/hour Trips % % % Travel
(min) (min)
Time (min)
Chakroborty &
105 89.98 10.02 0.00 4.06 4.06 13.10
Wivedi[6]
Nikolić &
94 91.91 8.09 0.00 4.32 3.14 11.32
Teodorović[22]
Arbex &
84 98.27 1.73 0.00 7.50 3.70 11.45
4 Cunha[9]
Jha &
- 97.17 2.83 0.00 - - 13.41
et al.[10]
DRL
72 98.01 1.99 0.00 8.06 4.13 11.55
(Trained for 4 routes)
DRL
72 98.01 1.99 0.00 8.06 4.13 11.50
(Trained for 6 routes)
Arbex &
81 98.20 1.80 0.00 7.60 4.25 11.08
Cunha[9]
5
DRL
76 99.04 0.96 0.00 11.3 5.58 11.24
(Trained for 6 routes)
Arbex &
79 98.20 1.80 0.00 8.22 5.94 11.55
Cunha[9]
Jha &
6 - 98.46 1.54 0.00 - - 12.61
et al.[10]
DRL
83 99.04 0.96 0.00 11.32 5.61 10.98
(Trained for 6 routes)

R EFERENCES [14] Dai, H., Khalil, E. B., Zhang, Y., Dilkina, B., & Song, L. (2017).
Learning combinatorial optimization algorithms over graphs. In
[1] Ceder, A., & Wilson, N. H. M. (1986). BUS NETWORK DESIGN. NIPS’17 Proceedings of the 31st International Conference on Neural
Transportation Research Part B-Methodological, 20(4), 331–344. Information Processing Systems (pp. 6351–6361).
[15] Nazari, M., Oroojlooy, A., Snyder, L., & Takac, M. (2018). Rein-
[2] Magnanti, T. L., & Wong, R. T. (1984). Network Design and Trans-
forcement Learning for Solving the Vehicle Routing Problem. In NIPS
portation Planning: Models and Algorithms. Transportation Science,
2018: The 32nd Annual Conference on Neural Information Processing
18(1), 1–55.
Systems (pp. 9861–9871).
[3] Valérie Guihaire and Jin-Kao Hao. “Transit network design and
[16] Kool, W., Hoof, H. van, & Welling, M. (2019). Attention, Learn to
scheduling: A global review”. In: Transportation Research Part A-
Solve Routing Problems! In ICLR 2019: 7th International Conference
policy and Practice. 42.10 (2008), pp. 1251–1273.
on Learning Representations.
[4] Mandl, C. E. (1980). Evaluation and optimization of urban public [17] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,
transportation networks. European Journal of Operational Research, A. N., . . . Polosukhin, I. (2017). Attention Is All You Need. ArXiv
5(6), 396–404. Preprint ArXiv:1706.03762.
[5] Baaj, M. H., & Mahmassani, H. S. (1991). An AI-based approach [18] Robert W. Floyd. “Algorithm 97: Shortest Path”. In: Commun. ACM
for transit route system planning and design. Journal of Advanced 5.6 (June 1962), pp. 345–. ISSN: 0001-0782.
Transportation, 25(2), 187–209. [19] Stephen Warshall. “A Theorem on Boolean Matrices”. In:J. ACM 9.1
[6] Chakroborty, P., & Wivedi, T. (2002). OPTIMAL ROUTE NETWORK (Jan. 1962),pp. 11–12. ISSN: 0004-5411.
DESIGN FOR TRANSIT SYSTEMS USING GENETIC ALGO- [20] Edsger W. Dijkstra. “A note on two problems in connexion with
RITHMS. Engineering Optimization, 34(1), 83–100. graphs”. In:Numerische Mathematik 1.1 (1959), pp. 269–271.
[7] Kidwai, F. A., Deb, K., Marwah, B. R., & Karim, M. R. (2005). [21] Kingma, D. P., & Ba, J. L. (2015). Adam: A Method for Stochastic
A GENETIC ALGORITHM BASED BUS SCHEDULING MODEL Optimization. In ICLR 2015: International Conference on Learning
FOR TRANSIT NETWORK. Representations 2015.
[8] Zhao, F., & Zeng, X. (2006). Optimization of transit network lay- [22] Nikolić, M., & Teodorović, D. (2013). Transit network design by
out and headway with a combined genetic algorithm and simulated Bee Colony Optimization. Expert Systems With Applications, 40(15),
annealing method. Engineering Optimization, 38(6), 701–722. 5945–5955.
[9] Arbex, R. O., & Cunha, C. B. da. (2015). Efficient transit network
design and frequencies setting multi-objective optimization by alter-
nating objective genetic algorithm. Transportation Research Part B-
Methodological, 81, 355–376.
[10] Jha, S. B., Jha, J. K., & Tiwari, M. K. (2019). A multi-objective meta-
heuristic approach for transit network design and frequency setting
problem in a bus transit system. Computers & Industrial Engineering,
130, 166–186.
[11] Lin, Q., Li, J., Du, Z., Chen, J., & Ming, Z. (2015). A novel multi-
objective particle swarm optimization with multiple search strategies.
European Journal of Operational Research, 247(3), 732–744.
[12] Bengio, Y., Lodi, A., & Prouvost, A. (2018). Machine Learning for
Combinatorial Optimization: a Methodological Tour d’Horizon. ArXiv
Preprint ArXiv:1811.06128.
[13] Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2017).
Neural Combinatorial Optimization with Reinforcement Learning. In
ICLR (Workshop).

Authorized licensed use limited to: Amrita School of Engineering. Downloaded on September 21,2023 at 03:37:01 UTC from IEEE Xplore. Restrictions apply.