2013 Seventh International Conference on Next Generation Mobile Apps, Services and Technologies
Road Hazard Detection and Sharing with Multimodal Sensor Analysis on
Smartphones
Fatih Orhan, P. Erhan Eren
METU Informatics Institute, Information Systems Dept.,
METU Campus, Ankara, Turkey
e112903@metu.edu.tr, ereren@metu.edu.tr
Abstract— The sensing, computing and communicating
capabilities of smart phones bring new possibilities for
creating smart applications, including in-car mobile
applications for smart cities. However, due to the dynamic
nature of vehicles, many requirements such as sensor
management, signal and image processing or information
sharing needs exist when developing a smart sensor-based
in-car mobile application. On the other hand, most in-car
applications generally employ single-modal sensor analysis,
which also yields limited results. Using the advanced
capabilities of smart phones, this study proposes a
framework with built-in multimodal sensor analysis
capability, and enables easy and rapid development of signal
and image processing-based smart mobile applications.
Within this framework, an abstraction for fast access to
synchronized sensor readings, a plugin based multimodal
analysis interface for signal and image processing
applications, and a toolset to connect to other users or
servers for sharing the results are provided built-in. As part
of this study, a sample mobile application is also developed
to demonstrate the applicability of the framework. This
application is used for detecting defects on the road, such as
potholes and speed bumps, and it automatically extracts the
video section and the image of the corresponding road
segment containing the defect. Upon such critical hazard
detection, the application instantly informs nearby users
about the incident. A good detection rate of speed bumps is
obtained in the performed tests, while the advantage of
automatic image extraction based on the multimodal
approach is also demonstrated.
Keywords-multimodal sensor analysis; mobile sensor;
mobile GIS applications; signal processing; pervasive
computing
I. INTRODUCTION
The convergence in mobile technologies enabled smart
phones to provide various uses due to their broad and
advanced sensing, computing and communication
capabilities. There are usually more than a dozen sensors
on a regular smartphone, in which location, motion and
camera sensors (as well as other sensors) are commonly
utilized to develop functional pervasive mobile
applications. One of the emerging areas in smart
applications is in-car mobile applications for smart cities,
where systems assist drivers in order to increase their
safety, comfort, economy, while also providing new
entertainment options.
Regarding the studies on road hazard detection, the
“Pothole Patrol” [1] study by Eriksson et al. which aims to
detect potholes on roads is considered as the first complete
and valid study for sensor usage in mobile in-car
applications. As part of a specially designed hardware, a
978-0-7695-5090-9/13 $26.00 © 2013 IEEE
DOI 10.1109/NGMAST.2013.19
Authorized licensed use limited to: Universiti Teknikal Malaysia Melaka-UTEM. Downloaded on September 27,2023 at 09:34:48 UTC from IEEE Xplore. Restrictions apply.
sequence of filters are applied including a speed filter
using the location and speed values from a GPS sensor,
and specifically a “Z-Peak” algorithm on accelerometer
values. Accordingly, the potholes are detected as a result
of these filters. Mohan et al. use sensors such as
accelerometer, GPS, GSM and microphone in their
Nericell [2] study. They focus on the detection of pothole,
speed bump, abrupt deceleration, and the sound of a horn.
For the pothole and speed bump detection, they apply the
“z-sus” algorithm and obtain successful results. A similar
study is conducted by Mednis [3] et al., where a smart
phone application is developed for detecting potholes and
extracting the profile of the road in real time. The
accelerometer values collected from the mobile device
sensor are used within a time window in different methods
such as comparing with a threshold value (Z-thresh),
taking the difference (Z-Diff) and taking the standard
deviation (STDev(Z)). The results show that the Z-Diff
algorithm yields the maximum performance. Astarita et al.
develop a system to detect potholes and speed bumps on
the road in their UNIquALroad [4] study. Their aim is to
assess the quality of the road, and they utilize a threshold
based filter on the acceleration values, obtaining %90
successful bump detection and %65 successful pothole
detection rates.
Besides the accelerometer sensor, cameras are also
widely utilized by in-car mobile applications for the
detection of obstacles on the road. KyuToku [5] et al.
develop a system to detect obstacles (such as pedestrian,
vehicle, pylon, box or ball) on the road by comparing the
frames captured by the camera against frames from past
recordings where there are no obstacles. First, the current
frame of the camera is mapped to one of the past frames
that were captured at the nearest position, then the road is
registered in both frames and the differences of the two
frames are extracted in order to detect the obstacles. The
results show that the obstacles are successfully detected at
a distance of less than 31 meters. Similarly Naito [6] et al.
also try to detect obstacles on the road by using the optical
flow method. Koch [7] et al. develop a system which
detects cracks and potholes on the road by using computer
vision techniques. The system operates in offline mode on
previously recorded videos, and performs the detection in
two steps. First, a set of predefined shapes are searched in
video frames and the system decides if there is an
irregularity or not. Then in the second step, the irregularity
type, the location, and the shape are extracted on the
image.
In brief, in-car mobile sensor applications generally
make use of accelerometer, magnetometer, GPS, and/or
camera sensors. However, most studies rely on a single
sensor; hence only single modal analysis is performed. The
56
 main reason to use a single sensor lies with the challenges
related to the development of multimodal sensor analysis
applications. These challenges may be listed as the
selection of hardware platform and its sensor drivers,
proper collection and synchronization of different sensors’
values, the integration of diverse libraries for additional
processing and also tools that provide connectivity for
sharing the results with other infrastructures and social
networks.
Our study focuses on these challenges and aims to
construct a framework which enables easy, fast and
flexible implementations of functional smart in-car
applications with high level complex analysis, and
provides various outputs including visual outputs. The goal
is to provide capabilities for real-time sensing, signal and
multimedia processing, and sharing of information with
other users and server applications. Hence, the proposed
framework provides an abstraction for easy access to
synchronized sensor readings, a plugin based analysis
interface, integration of signal and image processing
libraries, and a sharing toolset for information sharing.
With this framework, it is possible to implement
complex algorithms and develop analysis methods on the
collected and synchronized sensor values. On the other
hand, many problems which cannot be easily solved by the
single-modal approach may also be addressed in
multimodal analysis within this framework. In this sense,
the framework supports the output of one sensor analysis
to be used as an input to another sensor analysis.
A sample mobile application is also developed in this
study, in order to demonstrate the capabilities and the
feasibility of the developed framework. This application is
a multimodal sensor analyzer which detects obstacles on
the road, or critical road surface anomalies (i.e. pothole,
speed bump), extracts the video segment or photo of the
road section containing the hazard (“Fig. 1”) and sends the
hazard information to a central repository which distributes
the event to nearby users, in real time.
Figure 1. Sample application use-case
The system operates in two steps: first, the hazardous
road segment (pothole, speed bump) is detected by
analyzing the accelerometer and the vehicle’s velocity
(from GPS sensor) values. In the second step, the output of
the first step is utilized, and the visual imagery of the scene
is extracted as a video segment as well as an image, and
prepared for further image processing analysis (the image
processing is not performed in the scope of this study).
57
Authorized licensed use limited to: Universiti Teknikal Malaysia Melaka-UTEM. Downloaded on September 27,2023 at 09:34:48 UTC from IEEE Xplore. Restrictions apply.
Once a hazard is detected, the system has the capability to
inform nearby drivers (as geographic position) in order to
prevent accidents or undesirable events, by providing the
hazardous event detail as well as a screenshot image.
Experiments are conducted within the study to measure the
performance of the developed application, and the results
are exhibited.
Hence, the contributions of this study may be listed as:
1) a multimodal sensor framework 2) a sample application
demonstrating the benefits of multimodality and
serviceability of information sharing within a connected
world.
Section II includes the developed framework. In
Sections III and IV, the sample implementation utilizing
the framework and the experiment results are presented,
respectively. Finally, the conclusion is given in Section V.
II. PROPOSED FRAMEWORK
The goal of this study is to develop a mobile
multimodal sensor analysis framework which provides real
time sensing, multimedia processing and communication
capabilities. Although various smart mobile applications
may be developed with this framework, the core features
to be provided are selected as real-time synchronized
sensing, plugin based analysis interfaces, signal/image
processing capabilities and information sharing. The
framework is developed on the Android1 platform due to
its flexible and open architecture. The framework contains
three main components: “Sensing Component”, “Analysis
Component” and “Sharing Component” (“Fig. 2”).
Figure 2. Multimodal sensor analysis framework
architecture
A. Sensing Component
The sensing component provides an abstraction on
sensor devices, and thus facilitates access to raw sensor
values. Using this component, the developed application is
freed from dealing with the details of sensor device
registrations, malfunctions of device, or erroneous data.
Most importantly, this component enriches the sensor
values by synchronizing them and also provides the
developer the raw, aggregated and/or sampled values of
the sensors via simple interfaces, hence leading to easy but
accurate access to different sensor values when required as
part of the multimodal analysis. Another feature of the
sensing component is to reorient accelerometer values
prior to dispatching the sensor values (if developer selects
it), in order to get sensor readings already oriented to the
world coordinate system when needed.
To accomplish all of these, each sensor reading is
performed in a separate process, and all the sensors are
1
http://developer.android.com/index.html
 firstly globally time stamped as they are read from the
underlying operating system. Thus, it is possible to extract
the exact values of different sensors at any specific time
when needed. Within the separate threads, the aggregation,
sampling and reorientation operations are also performed.
Moreover, a logger interface is also provided within the
framework in order to easily log sensor values to a file.
Thus, the sensing component includes an interface defined
as “ISensorListener” to accomplish all of these tasks, and
as the developer implements this interface, the sensor
values are dispatched to the implementation.
For the camera sensor, the framework provides the
interface “ICameraPreviewListener”. With this interface, it
is possible to start/stop video recording, get the video file
after recording the camera sensor and split the file into
segments when start and end times are specified.
Alternative to providing the full video, the framework is
also able to provide the camera input frame by frame, thus
enabling processing of images in real time. Frames may be
provided as byte array or bitmap images.
B. Analysis Component
This component mainly provides interfaces that are
compatible with the output of the Sensing Component.
Thus, the values collected from the sensors are utilized
using these interfaces, and the analysis modules are easily
developed.
The signal processing in the analysis component
provides an abstract interface “ASensorAnalyzer” which
implements “ISensorListener” interface and provides easy
and effective analysis of collected sensor values. The
interfaces are flexible and easy to implement. Moreover,
the analysis component incorporates the use of image
processing libraries such as OpenCV2 .
The sensing component provides the frames captured
from the camera, and the developer may directly use the
OpenCV methods to apply image processing techniques on
these captured frames within the implementation of the
abstract class. Moreover, the infrastructure is integrated
with the FFmpeg 3 library, a software library for
manipulating multimedia resources, and the developer may
also utilize its functionality directly, without any further
integration.
C. Sharing Component
The applications developed with the framework may
easily be connected to a central application or
communicate directly with social networks. Thus, it is
possible to save all the records in a central repository for
further processing, and at the same time inform other users
about the outcomes of the sensor analysis, as well as get
information from other users. This component is based on
the client-server architecture and provides the following
features:
• A web service interface from server application
• An advanced upload service,
• A push-based messaging service
• Social networks integration
Due to the mobile and dynamic environment, the use of
push mechanism is strongly recommended for mobile
2
OpenCV Homepage, http://opencv.willowgarage.com
3 coordinate system will not be usually aligned with the
FFMPEG Homepage, http://ffmpeg.org/
58
Authorized licensed use limited to: Universiti Teknikal Malaysia Melaka-UTEM. Downloaded on September 27,2023 at 09:34:48 UTC from IEEE Xplore. Restrictions apply.
applications. The framework provides this mechanism as a
reusable component. Using this mechanism, the server
may efficiently communicate with the mobile devices,
using minimum resources.
III. SAMPLE APPLICATION OF THE FRAMEWORK
The main goal of the sample implementation is to
develop a multimodal sensor analysis mobile application in
order to demonstrate the applicability of the framework
and its advantages. The first sensor analysis is conducted
on the accelerometer and GPS data, and the second
analysis is conducted based on the result of the first
analysis, on the video file captured by the camera. This
way, instead of performing image processing on each
frame, the application uses the result of the first analysis in
order to decide which part of the sensor values to be
utilized in the second analysis.
The mobile application is designed as an in-car smart
application, to be used while on the road by attaching the
mobile device to the front windshield. The main goal is to
automatically detect hazardous events on the road while
driving, by analyzing the variations on accelerometer
values and capturing the imagery of the scene upon the
detection of a hazard. The application would also get
warnings from the system (via other drivers) and also
inform nearby drivers about the events that are newly
captured (“Fig. 3”). This information includes the exact
location, the severity of the hazard, and the detected
obstacle type (where possible), and image or video
recording of the incident encountered.
Figure 3. Mobile Application Usage Scenario
A. Mobile Application
Mounting the mobile device correctly to the vehicle is
a very important step in order to obtain valid sensor values.
The device should be tightly fixed to the vehicle in order to
accurately record the motion of the vehicle. For this
reason, it is a good idea to use a docking station or a
mobile device holder (as in “Fig. 4(a)”) attached to the
front window of the vehicle, and the mobile device be
fastened tightly to the holder. This operation is necessary,
otherwise rattling of the mobile device’s accelerometer
sensor produces incoherent values generated by the motion
of the mobile device itself, and this leads to erroneous
results.
However, fixing the mobile device is not adequate to
directly process sensor values since the mobile device’s
 vehicle’s coordinate system. For this reason, it is necessary
to reorient the mobile device coordinate system. The
reorientation operation is performed by the framework
itself, by applying Euler’s angle. The XYZ rotation
sequence is applied to accelerometer values, as explained
in [4].
(a) (b)
Figure 4. (a) Mobile application usage in car (b) Re-
oriented accelerometer axes
This way, the Z axis always shows the motion relative
to the gravity direction (Fig. 4 (b)), the Y axis the front and
backward motion, and the X axis side motions, according
to the world coordinate system. For all incident detection
algorithms, reoriented accelerometer values are utilized.
B. Road Obstacle Detection
Manual analysis of the accelerometer values shows that
each type of event has its own unique characteristic in
terms of accelerometer values change over time, as shown
in “Fig. 5". This study focuses on two types of analysis:
pothole and bump detection. Different sensor analyses are
implemented on accelerometer values for each event type.
Figure 5. Accelerometer values for car’s different
motions: (a) smooth road (b) pothole (c) speed bump (d)
abrupt deceleration
The pothole and speed bump detection is performed
within a time window of sensors value. First, the velocity
is acquired from the GPS sensor and is filtered out for the
values between 10 km/h (vmin) and 60 km/h (vmax). Then
the time window is calculated (as inversely correlated)
according to velocity, in equation (1). C is a coefficient
that is defined based on the vehicle characteristics and
Wmax is the maximum upper bound of the window. Wmax is
a pre-calculated value found as “36” (as a result of the
59
Authorized licensed use limited to: Universiti Teknikal Malaysia Melaka-UTEM. Downloaded on September 27,2023 at 09:34:48 UTC from IEEE Xplore. Restrictions apply.
calculation of the minimum speed of 10 km/h of a vehicle
passing an average of 2m long speed bump with
accelerometer sensor collection frequency at 40 Hz).
(1)
The time window usage is necessary because the same
hazard produces different number of samples for different
crossing speeds. Thus, for correct detection, only the
samples belonging to the hazard should be analyzed.
The accelerometer values are collected within a time
window (W) which is defined according to the instant
velocity (v) of the vehicle and the largest difference (z-
diff) along with Z axis values is calculated within that time
window. For pothole detection, if z-diff value exceeds
some predefined threshold (Tb), the detection is considered
as successful (equation (2)).
(2)
For the speed bump detection, the sensor value changes
in “Fig. 5 (c)” are analyzed. According to this pattern, the
sensor values exceed several thresholds sequentially and
with a minimum interval between sequences. As depicted
in Fig. 6, first, exceeding a maximum threshold (z-max1),
then the minimum threshold (z-min1), then a second
maximum threshold (z-max2), and lastly another minimum
threshold (z-min2) indicates a crossing of the vehicle over
a speed bump.
Figure 6. Minimum duration calculation in speed bump
detection
The thresholds z-max and z-min are calculated
according to equations (3), (4), and (5). First, z-avg is
calculated as the average of z values within the time
window. Then a deviation value is computed according to
the velocity (v) of the vehicle multiplied by a coefficient,
defined according to the vehicle properties. Then z-max
and z-min are calculated according to equations (4) and (5)
respectively.
 (3) distance is found in equation (7). After finding the number
of sensor values to be read backwards, the total time
(4) passed (∆Td) in reading n sensor values is calculated in
equation (8).

(5)
(7)

The interval (∆t) between z-max1 and z-max2 is very

important for the detection of a speed bump, since a small
interval value is an indication of other kinds of (8)
irregularities (such as pothole, or any other sudden change
of road surface), and a too large interval value is a
probable indication of two consecutive potholes. Thus, the
time interval (∆t) is compared against the threshold value (9)
T, and the bump is detected according to the equation (6).
Since the sensor values and the video are synchronized,
the correct location of tshot for the image to be extracted
from the video is calculated according to equation (9).
(6)

An important point in bump detection is the fact that an

increase in vehicle’s velocity generates a large fluctuation
in the values of the accelerometer sensor. For this reason,
the threshold values Tmax and Tmin used in the detection of
hazards is adjusted according to the velocity of the vehicle.
Thus the sequence of operations performed in speed bump
detection is as follows: 1) discard sensor values of too low
or too high velocities, 2) determine the thresholds based on
time and velocity of the vehicle, 3) detect consecutively
twice the exceeding of thresholds as Tmax and Tmin and 4)
ensure that a period of time passes between z-max1 and z-
max2.
C. Image Extraction
A distinctive feature of this study is the implementation
of the multimodal analysis approach. The study enables the
analysis of sensor values in one modality based on the
results of sensor values of a different modality. With this
approach, the video, which is recorded simultaneously
while detecting the hazards, is analyzed and the
“important” parts of the video, which contain the hazard
scenes, are extracted by using the detection results of the
analysis on the accelerometer sensor values.
Since image processing is a time and resource
consuming operation, instead of analyzing the whole video
from start till the end, only the video parts which have to
be focused and which are likely to contain a hazard, are
extracted by the multimodal analysis. The video parts and
frames are extracted with this method and prepared for
further processing (image processing on extracted frames
is not conducted as a part of this study).
For the video and frame extraction, the following
method is applied: for a detected road hazard , the video
that has been recorded is analyzed, and a video section
starting from 10 meters ahead of the hazard until the
hazard, and an image from 5 meters ahead of the hazard is
extracted automatically (“Fig. 7”). In order to achieve this,
the exact video location which matches the required
distances before the hazardous road segment is identified.
For image extraction, we have selected the best
distance for a good image before the bump speed as 5
meters. Assuming X0 = 5m, the number of sensor values
(n) that should be considered backwards to span that
Figure 7. Image of nine different detected speed bumps,
extracted automatically
D. Sharing with Other Drivers
Using the Sharing Component of the framework, the
application is able to warn the nearby users about a hazard
on the road. Hence, upon a detection of a road hazard, the
application immediately creates an event which consists of
the time, location, hazard type and hazard severity data.
This event is sent to a central server (if the device has
connection). The server then runs a quick geo-location
search to locate the active drivers nearby the event and
sends a notification about the event to these drivers, using
push mechanism.
Once the event is received by the vehicle, the
application checks whether that vehicle motion is headed
towards the event or not. There, we simply check whether
the direction between the event location and the vehicle
motion direction. If the direction is towards the event, the
driver is warned about it, when the distance is less than a
threshold (again calculated according to the velocity of the
vehicle). The warning may be presented by visual or audial
alerts, based on the user preferences.
IV. EXPERIMENT RESULTS
In order to evaluate the performance of the mobile
application, a route in our university campus is selected
and several driving tests are conducted on this route. In

60

Authorized licensed use limited to: Universiti Teknikal Malaysia Melaka-UTEM. Downloaded on September 27,2023 at 09:34:48 UTC from IEEE Xplore. Restrictions apply.
 order to build the ground truth for potholes and speed all. This shows that the algorithms are too sensitive, and
bumps, the correct locations are marked manually before needs to be adapted according to different conditions and
testing, by travelling the entire route. The results are different shapes of obstacles.
collected from the test runs and compared with the ground
truth values, and they are evaluated in terms of recall and V. CONCLUSION
precision values. In this study, a multimodal sensor analysis framework
is developed for in-car mobile applications for performing
TABLE I. EVALUATION RESULTS analysis on real-time synchronized sensor values. The
Speed Bump Counts and Recall and Precision Results mobile application developed within this framework
Test No Ground True False False Recall Precision detects the potholes and speed bumps on the road, and
Truth Positive Positive Negative (%) (%) successfully extracts the image and video section
1 11 8 3 3 73 73 corresponding to this road segment automatically. This
2 12 10 3 2 83 77 extracted section is prepared for further image processing.
3 10 9 1 1 90 90
4 11 7 0 4 64 100
The detection is shared over a central application with
5 11 9 2 2 82 82 other drivers, which provides a quick awareness about
6 11 8 0 3 73 100 traffic events.
7 9 7 1 2 78 88 With the methods employed in the study, the
8 4 3 0 1 75 100 advantages of multimodal analysis over single-modal
9 11 11 4 0 100 73 analysis are exhibited. As future work, it is planned to
10 11 10 2 1 91 83 increase the robustness of the algorithms, to focus on more
11 10 8 1 2 80 89 complex analysis using different sensors (such as
12 11 9 0 2 82 100
13 11 9 0 2 82 100
microphone, magnetometer, or proximity), and to explore
14 11 10 0 1 91 100 social aspects of utilizing the outcomes of the sensor
15 9 7 0 2 78 100 analysis for smart city applications.
16 10 9 0 1 90 100
17 12 10 0 2 83 100 REFERENCES
18 11 10 0 1 91 100 [1] Eriksson, J., Girod, L., Hull, B., Newton, R., Madden, S.,
19 11 9 0 2 82 100 Balakrishnan, H.: The Pothole Patrol: Using a Mobile Sensor
20 13 11 0 2 85 100 Network for Road Surface Monitoring. In: MobiSys'08. pp. 29-
21 16 13 2 3 81 87 39, 2008
22 10 7 0 3 70 100 [2] Mohan P., Padmanabhan V. N., and Ramjee R., “Nericell: using
23 12 9 0 3 75 100 mobile smartphones for rich monitoring of road and traffic
24 12 11 1 1 92 92 conditions,” in Proceedings of the 6th ACM conference on
25 11 9 0 2 82 100 Embedded network sensor systems, ser. SenSys ’08. New York,
AVG 10.84 8.92 0.80 1.92 82 93 NY, USA: ACM, pp. 357–358, 2008
[3] Mednis A., Strazdins G., Zviedis R., Kanonirs G., Selavo L.,
Real time pothole detection using android smartphones with
Test drive durations are 20 minutes on average and a
accelerometers. Distributed Computing in Sensor Systems and
total of 25 drives are conducted within the test. The results Workshops, International Conference on 0, 2011
show a recall value of 82% and a precision of 93% (Table [4] Astarita V, Caruso M.V., Danieli G., Festa D.C., Giofrè V. P.,
I) for successful bump detection. A detailed manual Iuele T., Vaiana R., A mobile application for road surface quality
analysis on the results is also conducted, and two speed control: UNIquALroad, Procedia - Social and Behavioral
bumps are identified for which the algorithms mostly Sciences 54, 2012
failed to detect, thus resulting in false negative. With the [5] Kyutoku, H., Deguchi, D., Takahashi, T., Mekada, Y., Ide, I.,
manual inspections on these two speed bumps, it has been Murase, H.: On-road obstacle detection by comparing present
and past in-vehicle camera images. In: Proc. 12th IAPR Conf. on
observed that they are physically different in shape Machine Vision Applications, pp. 357-360 (2011).
compared to others, such that one side of the speed bump [6] T. Naito, "The Obstacle Detection Method using Optical Flow
slope was lower than a regular one, thus they produce Estimation at the Edge Image," IEEE conference on Intelligent
lesser variations in accelerometer values than expected. Vehicles Symposium, pp. 817-822, 2007.
For this reason, the algorithms fail to detect the first [7] Koch, C. and Brilakis, I., ‘Pothole detection in asphalt pavement
threshold exceeding, hence fail to detect the speed bump at images’, Advanced Eng. Informatics 25(3), 507–515, 2011

61

Authorized licensed use limited to: Universiti Teknikal Malaysia Melaka-UTEM. Downloaded on September 27,2023 at 09:34:48 UTC from IEEE Xplore. Restrictions apply.