Real-time Application of YOLO Models in a Pothole

Detection System

Group members:
Donatus Darren Khusuma XI.2/10

Parrish Sebastian Mayhew Lee XI.1/26

 Foreword
The amount of potholes that Indonesia has to deal with gradually increases over the

years, this is largely suspected to be due to the increased rainfall from climate change. These

potholes cause many problems for people on the road since repairing damages caused by these

potholes can be worth a hefty amount of money. In some cases, these potholes can cause drivers

to lose control over their vehicles and cause a large disturbance in the area by crashing into

nearby objects. In our current age, the usage of AI for industrial usage and operations has

become a large trend in society with object detection, natural language processing, and neural

networks. Thus, the following events had sparked some attention in the topics of “potholes” and

“object detection”. For this reason, this research is dedicated towards the real-time usage of an

AI model for detecting potholes and alerting them to whomever it may concern.

The people working on this study would like to give their gratitude and appreciation

towards the following people who have guided and provided their assistance towards this project

be it directly or indirectly:

- Hendra Kusumah, Mohamad Riski Nurholik, Catur Putri Riani, Ilham Riyan Nur

Rahman, for their valuable study and recommendations within their research article

about “Deep Learning for Pothole Detection on Indonesian Roadways”

- Hendra Kusumah and Muhammah Suzaki Ahran, for their valuable guidance and

experience in regards to helping to determine how to execute the project

- Sir Nicolaus Adityoso Putro, for his guidance in the formation of this project
 It is with great pleasure that we have been allowed to work on this project and it is with

hope that this project will bring more light onto road disturbances and reduce the amount of time

it takes for the potholes to be repaired by local authorities and allow more accidents involving

potholes to be avoided. In addition, we will also be testing the performance of the latest AI

models to determine which model is most suitable for real-life application.

 Table Of Contents

Foreword....................................................................................................................................................... 2

Table Of Contents.......................................................................................................................... 3

CHAPTER 1 Introduction............................................................................................................ 4

1.1 Background.......................................................................................................................... 4

1.2 Problem statement................................................................................................................5

1.3 Research Questions..............................................................................................................5

1.4 Research Objectives.............................................................................................................5

1.5 Benefits of Research............................................................................................................ 5

CHAPTER 2 Basic Theory........................................................................................................... 6

2.1 Potholes................................................................................................................................6

2.1.1 Definition.................................................................................................................... 6

2.1.2 Pictures........................................................................................................................6

2.1.3 Cause of Potholes........................................................................................................6

2.1.3.1 Flooding and Rain..............................................................................................6

2.1.3.2 Road Surface Cracks..........................................................................................7

2.1.3.3 Landslides/Earthquakes.................................................................................................. 7

2.1.4 Why does Indonesia have a lot of potholes?...............................................................7

2.2 YOLO.................................................................................................................................. 7

2.2.1 YOLOv5..................................................................................................................... 7

2.2.2 Usage in Study............................................................................................................ 7

 2.3 Pothole Identification...........................................................................................................8

2.3.1 Artificial Intelligence.................................................................................................. 8

2.3.1.1 Advantage.......................................................................................................................8

2.3.1.2 Drawbacks...................................................................................................................... 8

2.3.2 Infrared Sensor............................................................................................................8

2.3.2.1 Advantage.......................................................................................................................8

2.3.2.2 Drawbacks...................................................................................................................... 8

2.3.3 Ultrasonic Sensor..................................................................................................................... 8

2.3.3.1 Advantage.......................................................................................................................8

2.3.3.2 Drawbacks...................................................................................................................... 9

2.3.4 Lidar Sensor............................................................................................................................. 9

2.3.4.1 Advantage.......................................................................................................................9

2.3.4.2 Drawbacks...................................................................................................................... 9

2.4 Hypothesis............................................................................................................................9

CHAPTER 3 Methodology..........................................................................................................10

3.1 Research Method......................................................................................................... 10

3.2 Variables.......................................................................................................................10

3.3 Materials and Apparatus.............................................................................................. 10

3.3.1 Materials............................................................................................................. 10

3.3.2 Apparatus............................................................................................................ 10

3.4 Work Procedures.......................................................................................................... 10

3.4.1 Training the Model..............................................................................................10

 3.4.2 Creating the Alert Signal program with a built-in timer.....................................10

3.4.3 Deploying the Prototype..................................................................................... 11

3.5 Research Methodology.................................................................................................................... 11

3.5.1 Flowchart................................................................................................................................11

Figure 4 Research Methodology Flowchart.................................................................................... 11

3.5.2 Step-by-step............................................................................................................................11

CHAPTER 4 Closing Statement.................................................................................................13

REFERENCES........................................................................................................................................... 14

Information............................................................................................................................................ 14

Pictures................................................................................................................................... 15
 CHAPTER 1 Introduction
1.1 Background
Recently at the time of this research, there was a pothole in Jalan Gurila that caused a

major disturbance in traffic[1]. Jalan Gurila in Medan Perjuangan District, Medan City, was

reportedly badly damaged with many large holes causing motorists to fall frequently. The holes

are located near the intersection of Jalan Gurila and Jalan Williem Iskandar, with several holes

measuring 15-20 cm. The condition of this damaged road is around 20 meters deep.

Residents have complained about this condition for years, and the situation gets worse

during the rainy season. Some motorists were forced to pass slowly on the side of the road to

avoid potholes covered by puddles of water. A resident, Marni, expressed his concern about the

dangers of the damaged road, considering several accidents and casualties at that location. Marni

urged the government to repair Jalan Gurila to prevent further accidents immediately.

Similarly, several other local complaints from people in Indonesia relate to the problem

of potholes around the area:

- Bus driver[2] complains about the need for new renovation and fixing of potholes on the

roads in Terminal Bubulak(16/01/2024)

- Civilian driver[3] complains about big pothole near Cicadas Market,

Bandung(15/01/2024)

- Aden Bajaj[4], an Indonesian actor and comedian protests about potholes on the roads in

Depok and posts about them to the vice mayor of Depok and his secretary on Instagram,

his post was answered and the pothole was fixed on the very next day(23/01/2024)
 According to the head of DKI Jakarta Highways Agency[5], it was reported that

approximately 21,442 potholes in Jakarta were fixed during December 2022 up until early March

2023.

In the current 21st century, AI has become a large trend in today's society. AI is

implemented for tasks involving computer vision, neural networks, and natural language

processing[6]. Many contents from social media such as YouTube and Instagram alike have

tutorials on making real-time object detection using Python.

Following this trend, the use of object detection has been officially used for detecting

potholes in a study by 4 students from the University of Raharja[7]. Though there were several

setbacks within the study that showed that there are still some weaknesses in using AI for

detecting potholes such as the model accidentally detecting objects that aren’t potholes like fallen

branches, leaves, repaired potholes, etc.

1.2 Problem statement

Thousands of potholes go undetected each year and can cause numerous difficulties in

driving. Potholes are road hazards for motorists and municipalities, causing vehicle damage,

traffic disruptions, water damage, etc[8]. Potholes in Indonesia are often the result of weather

conditions such as rain which can wear down the infrastructure of the road itself making it more

susceptible to breaking down. Since these potholes go undetected for long periods, they pose

many different driving problems for drivers.

1.3 Research Questions
● How well does the AI model perform the task of detecting and alerting others on potholes

in real-life?

● Which of the AI models used is the most suitable for real-life application?

● What kind of settings should the AI model have to increase overall performance?

1.4 Research Objectives

● To test the capabilities of the Pothole Detection System in alerting others about potholes.

● To determine which of the AI is most suitable for real-life applications

● To determine the type of settings required for an increase in the model’s performance

1.5 Benefits of Research

The benefits of this research consist of:

1. Early detection: Alerting drivers of upcoming potholes

2. Have a better understanding of where potholes are most prevalent.

3. Reduced accidents: By accurately and efficiently detecting potholes, your research could

lead to faster repairs, minimizing the risk of accidents caused by potholes.

4. Improved data collection and analysis: Contribution to better data collection and analysis

methods for pothole detection and road maintenance.

 CHAPTER 2 Basic Theory
2.1 Potholes
2.1.1 Definition
According to Oxford Languages[9], the definitions for potholes are

1. a deep natural underground cave formed by the erosion of rock, especially

by the action of water.

2. a depression or hollow in a road surface caused by wear or subsidence.

In this research, the second definition of the term pothole will be the main focus

of this entire project.

2.1.2 Pictures

Figure 1 Figure 2 Figure 3

POTHOLE ON THE ROAD WORKERS REPAIRING CAR GETS STUCK ON

POTHOLE IN JOGJAKARTA POTHOLE

2.1.3 Cause of Potholes

There are several factors that contribute to the formation of potholes such

as floods, rains, road surface cracks, landslides, etc.[10]. Each of these factors

contributes to the deformation of the ground. Potholes form because of water in

the underlying soil that weakens the support acting on the road surface as well as

traffic driving overhead which can slowly displace the surface deeper into the

ground.
 2.1.3.1 Flooding and Rain
According to Safitri(2023)[11], flooding and rain alike cause damage to

roads due to a process commonly known as erosion. The waters that flow from

the road to the drain take away the weak asphalt from the roads and bring them

along, additionally, the water that seeps through the road and inside the drains can

destabilize the ground.

2.1.3.2 Road Surface Cracks

Common causes of road surface cracks are overloading, seepage, improper

or poor road surface drainage, lack of proper road maintenance, lack of proper

design, adverse climatic conditions, and some other factors[12]. Road surface

cracks can contribute to weakening the pavement surface, allowing more water to

seep into the cracks and get absorbed into the underlying soil.

2.1.3.3 Landslides/Earthquakes
Similar to road surface cracks, landslides, and earthquakes alike cause

damage to road infrastructure by liquefying the soil, causing debris, stones,

etc[13]. to fall on the road thus causing more damage, as well as surface ruptures.

2.1.4 Why does Indonesia have a lot of potholes?

According to a news article from viva.co.id[14] and Statista.com[15].

Indonesia has a massive problem with flooding, as can be seen on the roads where

the floods can reach as high as 50 cm in just a day, and from the number of flood

incidents that occurred in Indonesia during the years 2016-2022. Additionally, Indonesia

has a population of almost 279 million people[16] at the time of this research, it has also

been suspected that a larger population would also cause an increase in the amount of

traffic on the roads, particularly in overpopulated cities such as Jakarta. As such, there is
 bound to be many road cracks due to all the cars driving overhead. Finally, Indonesia

experienced an average of 851 landslides per year from 2016-2022[17].

The following facts along with the information provided in 2.1.3.1-2.1.3.3 show

that Indonesia has a multitude of factors that contribute significantly to the amount of

potholes that are formed on the road.

2.2 Pothole Identification

In order to identify potholes, several methods can be employed. However, each of these

methods has its drawbacks and advantages over other methods. As such, the most simple and

effective method with the highest prospects of succeeding was used for this research.

2.2.1 Artificial Intelligence

Artificial intelligence in object detection will be the most helpful in our research,

including identifying potholes. Object detection is a computer vision task that is used to

detect which objects are located where within an image[18]. These are the advantages

and drawbacks of using Artificial Intelligence[19] :

2.2.1.1 Advantage
1. Reduced human error

2. Pattern recognition

3. Automating repetitive tasks

2.3.1.2 Drawbacks
1. Requires data to function

2. False positives and negatives

3. Extensive labor and costs

2.2.2 Infrared Sensor
An infrared sensor (IR sensor) is a radiation-sensitive optoelectronic component

with spectral sensitivity in the infrared wavelength range[20]. IR sensors are often used

as motion detectors which is a suitable method for detecting potholes. These are the

advantages and drawbacks regarding the use of Infrared Sensors[21] :

2.2.2.1 Advantage
1. Day and Night compatibility.

2. Low electrical usage.

3. Able to detect up to 5 metres.

2.2.2.2 Drawbacks
1. Dust, rain, and other weather-related phenomena can trigger sensors.

2. Limited range.

3. Requires camera.

2.2.3 Ultrasonic Sensor

An ultrasonic sensor is an instrument that measures the distance to an object using

ultrasonic sound waves [22]. An ultrasonic sensor uses a transducer to send and receive

ultrasonic pulses that relay back information about an object’s proximity. These are the

advantages and drawbacks of using Ultrasonic Sensors :

2.2.3.1 Advantage
1. Wide range of detection up to 16.5 meters.

2. Able to detect objects irrespective of color, material, or transparency

making them incredibly versatile.

3. Day and Night compatibility.

4. Designed for outdoor use with water-resistant components.

 2.2.3.2 Drawbacks
1. Not inherently waterproof

2. Intense rainfall can disrupt sound waves and affect accuracy

3. Accuracy is dependent on the speed of sound, which can vary with

temperature.
2.2.4 Lidar Sensor
Lidar sensors, short for Light Detection and Ranging, are devices that use pulsed

laser light to measure distances and create 3D maps of their surroundings [23]. They

work by emitting a beam of laser light and then measuring the time it takes for the light to

bounce back from objects in the environment. By knowing the speed of light, the sensor

can then calculate the distance to each object.

2.2.4.1 Advantage
1. High precision, can detect objects with vertical dimensions of 5-15 cm and
horizontal dimensions of 30-50 cm.
2. Fast Acquisition & Processing

3. Provides precise 3D mapping resulting in minimal weather independence

2.2.4.2 Drawbacks
1. Extremely high costs
2. Limited range is insufficient for applications requiring detection over long
distances
3. Requires Experience to Operate

2.3 YOLO
YOLO(You Only Look Once) is an object detection and segmentation model that was

developed by Joseph Redmon and Ali Farhadi [24]. Using YOLO, it is possible to identify

potholes in the singular frames of a video recorded in real-time with ease.

2.3.1 YOLOv5
According to roboflow.com[25] and sciencedirect.com[26], YOLOv5 is an object

detector to detect small objects. It is the fastest model compared to other object detectors.
 YOLOv5 is a model in the You Only Look Once (YOLO) family of computer vision

models. YOLOv5 is commonly used for detecting objects. YOLOv5 comes in four main

versions: small (s), medium (m), large (l), and extra large (x), each offering progressively

higher accuracy rates. This model was used within the study mentioned previously within

the background.

2.3.2 Previous Usage

The study of “Deep Learning for Pothole Detection on Indonesian Roadways”

had used the YOLOv5n model to detect potholes by training it against a dataset of

motorcycle vlogs moving at speeds of 20-30 km/h.

2.4 Raspberry Pi

Raspberry Pi is essentially a mini-computer packed with all the essential components - a

processor, RAM, storage, USB ports, HDMI output, and GPIO pins - but at a fraction of the size

and cost of a traditional desktop computer.

2.4.1 Raspberry Pi zero 2 W

Raspberry Pi zero 2 W is a single-board computer [27]. It is able to contain an

operating system and multitask with high performance, relatively low pricing, and ease of

use like every other Raspberry Pi module.

2.4.2 How to input a program into Raspberry Pi module

1. Connect your Raspberry Pi to a power source and ensure it is connected to the

internet.

2. Use a text editor to write your program code on your computer.

 3. Transfer the program code to the Raspberry Pi using methods such as SSH, FTP, or

a USB drive.

4. Once the program code is on the Raspberry Pi, you can run it using a terminal

window.

2.5 Hypothesis
If the system manages to detect potholes on the road with above 80% accuracy and alerts

the drivers in less than 1 second after identification at varying speeds of 10-60 km/h. Then it

shows that the system is ready for usage in various situations of traffic because the system has

proven its capability to detect potholes relatively accurately as well as alert others of the potholes

at different speeds.
 CHAPTER 3 Methodology
3.1 Research Method
First, the models will be trained using a dataset consisting of pictures of potholes

captured from varying distances in vehicles at varying speeds. Next, a prototype of the detection

system will be made by implementing the model into a program that also checks whether or not

any pothole identification has been made and then makes a signal if it has. Afterward, the

prototype will be put inside a single-board computer like the Raspberry Pi Zero and connected to

a dashcam. Finally, the prototype will be tested on the field by having it identify a total of 20

potholes and its performance will be recorded.

3.2 Variables
Control Dataset of potholes, Camera quality, Confidence Threshold of the models

Control Group YOLOv5nu model

Independent YOLO models (YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, YOLOv8x)

Dependent Pothole identification accuracy, Delay between identification and alert

signal

3.3 Materials and Apparatus

3.3.1 Materials
- YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, YOLOv8x, YOLOv5n model

- Dash Cam

- Raspberry Pi zero 2 W

- Dataset of potholes

3.3.2 Apparatus
- Ultralytics Hub
 - PyCharm

3.4 Work Procedures

3.4.1 Training the Model
1. Create a dataset of at least 1000 photos of potholes captured from vehicles with

speeds ranging from varying speeds and varying camera qualities.

2. Train the models using the potholes identified within the dataset for 20 epochs.

3. Cross-check and validate the identification of the potholes identified by the

trained model using the pothole.

3.4.2 Creating the Alert Signal program with a built-in timer

1. Create a program that contains the trained model and checks whether the model

has identified a pothole within its view. Once, the model has identified a pothole

on the road, create a signal

2. Input the program into a Raspberry Pi Zero 2 W and connect it to a camera that

the program can access.

3.4.3 Deploying the Prototype

1. Attach the prototype to the front of a vehicle and test its performance at varying

speeds and using different vehicles

2. Use different models each time the test is performed

3.5 Research Methodology
3.5.1 Flowchart

Figure 4 Research Methodology Flowchart

3.5.2 Step-by-step
1. Do research on Potholes

2. Collect Pictures for the dataset used to train the models & Program an alert signal

response with a built-in timer

3. Train the AI models to identify the potholes in the dataset

4. Cross-validate the data from the AI model to eliminate discrepancies

5. Check whether both the model and the program are ready for use

6. Test the prototype(Using different models and vehicles at varying speeds) on the

field

7. Record the performance of the prototype

 CHAPTER 4 Closing Statement
The innovation to be made through this research is the practical usage of AI for detecting

and alerting potholes to others nearby. This innovation could speed up the repairs made on the

roads by alerting local authorities and giving opportunities for the drivers to avoid these potholes

when they aren’t paying attention. Additionally, this innovation can help to bring more statistical

data about the potholes in Indonesia which could be used to help others in their research on

potholes.
 REFERENCES
Information
[1] Sari, K. (2024, January 22). Banyak Ranjau Lubang di Jalan Gurila, Warga: Pengendara Sering Jatuh.
Detiksumut.

[2] Pratama, A. M. (2024, January 16). Tak Terawatnya Terminal Bubulak Bogor, Sampah Berserakan dan
Jalanannya Berlubang. KOMPAS.com.

[3] Aurellia, A. (2024, January 15). Lubang di Jalan Jakarta Kota Bandung Dikeluhkan Pengendara.
Detikjabar.

[4] Sumarni, & Pangesti, R. (2024, January 23). Aden Bajaj protes Jalan GDC Depok rusak parah, petugas
Auto Gercep perbaiki. suara.com.

[5] Haan, K. (2023, April 24). How businesses are using artificial intelligence in 2024. Forbes Advisor.

[6] ​Kusumah, H., Nurholik, M. R., Riani, C., & Rahman, I. R. N. (2023). Deep learning for pothole detection
on Indonesian roadways. Journal Sensi, 9(2), 175–186.

[7] Tobing, A. G. L. (2023, March 10). penanganan, perbaikan, jalan, lubang, curah, hujan, hotmix, coldmix,
beton, bina marga, rusak, galian, longsor, identifikasi, monitoring, alkal, satgas - Beritajakarta.id.

[8] Steven M. Lee, PC. (2019, July 23). A pothole can damage your vehicle, your passengers, and you |
Steven M. Lee, PC.

[9] Oxford Languages | the home of language data. (2022, August 26).

[10] THE CAUSES OF POTHOLES ON ROADS, THEIR EFFECTS AND CONTROL METHODS. (2023,
November 12). EngineeringAll.

[11] C, S. (2023, May 8). Pengaruh Banjir Terhadap Jalanan Rusak di Lampung | GEOTIMES. GEOTIMES.

[12] Road cracks | Types of road cracks - Road cracking reasons. (n.d.).

[13] Earthquake damage, danger & destruction. (2020, August 10).

[14] Ruqoyah, S. (2023, December 8). Diguyur Hujan Deras, Ini Jalanan di Jakarta yang Terendam Banjir.
VIVA.

[15] Indonesia: number of floods 2022 | Statista. (2023, November 22). Statista.

[16] Indonesia population (2024) - Worldometer. (n.d.).

 [17] Indonesia: number of landslides 2022 | Statista. (2023, May 2). Statista.

[18] Boesch, G. (2024, February 14). Object Detection in 2024: The Definitive guide. viso.ai.

[19] Duggal, N. (2024, February 16). Advantages and Disadvantages of Artificial intelligence [AI].
Simplilearn.com.

[20] Infrared sensor - IR sensor | Sensor division knowledge. (n.d.).

https://www.infratec.eu/sensor-division/service-support/glossary/infrared-sensor/

[21] Manzolillo, N. (2023, September 11). IR Sensors: What is an Infrared Sensor and How to Use Them?
ButterflyMX® - Official Site | Video Intercoms & Access Control.

[22] Carlson, C. (2023, March 1). How ultrasonic sensors work. MaxBotix.

[23] What is lidar? (n.d.). https://oceanservice.noaa.gov/facts/lidar.html

[24] Ultralytics. (2024, February 3). Home. Ultralytics YOLOv8 Docs.

[25] YOLOv5 Object Detection Model: What is, How to Use. (n.d.).

[26] Zaidi, S. S. A., Ansari, M. S., Aslam, A., Kanwal, N., Asghar, M. N., & Lee, B. (2022). A survey of modern
deep learning based object detection models. Digital Signal Processing, 126, 103514.

[27] Ltd, R. P. (n.d.). Buy a Raspberry Pi Zero 2 W – Raspberry Pi. Raspberry Pi.

[28] Bučko, B., Lieskovská, E., Zábovská, K., & Zábovský, M. (2022). Computer Vision Based Pothole
Detection under Challenging Conditions. Sensors, 22(22), 8878.

[29] ​MotorEasy. (n.d.). Pothole claim: How to report pothole damage | MotorEasy.

Pictures
Figure 1 Furqan, M. F. (2023, June 13). Download Yogyakarta, Indonesia - March 20, 2023 - Two workers
repairing asphalt on Mayor Suyotomo street, near Malioboro. Pothole repair. Vecteezy.

Figure 2 Nagesh, A. (2015, January 10). Britain’s pothole misery claims another victim as Vauxhall Corsa
tumbles into crater. Express.co.uk.

Figure 3 Nagesh, A. (2015, January 10). Britain’s pothole misery claims another victim as Vauxhall Corsa
tumbles into crater. Express.co.uk.

Figure 4 Custom - Made by DONATUS DARREN KHUSUMA