SoK: Adversarial Analysis of Autonomous
Vehicular Perception Systems
Arkajyoti Mitra
dept. of Computer Science and Engineering
University of Texas at Arlington
email address or ORCID
Abstract—Autonomous driving has made significant strides,
marking a transformative shift in the automotive industry. From
the conceptual stages to tangible reality, autonomous vehicles are
now commercially available to the global population, promising
safer, more efficient transportation. The main driving force
behind it has been a well-built perception system. With the
aid of sensors such as LiDAR, radar, and cameras, autonomous
driving has become a reality on roads, fundamentally reshaping
our perception of transportation. However, amidst this progress,
with these technological advances, the attack surfaces have
also inevitably grown, raising concerns about the security of
these vehicles and prompting ongoing research efforts. While
numerous works have focused on revealing the vulnerabilities of
autonomous driving, the literature regarding these vulnerabilities
lacks systematic organization. This paper aims to systemati-
cally organize knowledge surrounding the perception systems
of autonomous vehicles, delving into intricate details to under-
stand their functioning and limitations. In this taxonomy, we
examine several proposed attacks on these perception systems
and categorize them based on their feasibility, vivid impact,
and limitations. We further discuss existing defenses against
such adversarial attacks on the perception systems. Finally,
through meticulous analysis, we propose new research directions
with the hope of future endeavors, emphasizing the need to
enhance security measures, refine decision-making algorithms,
and explore novel methods for robust perception in varied
environmental conditions.
I. I NTRODUCTION
Autonomous vehicular perception stands as the cornerstone;
the driving force behind the commercialization of self-driving
cars lies in the promise of enhanced safety, efficiency, and
convenience. Central to this transformative vision is the devel-
opment of perception techniques that need to perceive, inter-
pret, and respond to their dynamically changing environment
in real-time.
With sophisticated machine learning algorithms such as
neural networks, autonomous vehicular perception enables
vehicles to navigate complex traffic scenarios, anticipate po-
tential hazards, and make split-second decisions with accuracy
that surpasses human capabilities. Moreover, these perception
systems hold the potential to mitigate the human errors re-
sponsible for the majority of traffic accidents, thereby saving
countless lives and reducing the economic costs associated
with road accidents. Autonomous vehicles can construct de-
tailed and dynamic representations of their environment in
1
Habeeb Olufowobi
dept. of Computer Science and Engineering
University of Texas at Arlington
email address or ORCID
real-time by integrating data from sensors such as cameras,
LiDAR, radar, and ultrasonic sensors.
Autonomous vehicular perception opens up new frontiers
in mobility and urban planning. Self-driving cars can reduce
congestion and pollution while optimizing the use of existing
infrastructure. Moreover, they promise to enhance accessibil-
ity for individuals with disabilities and elderly populations,
unlocking newfound freedom and independence.
However, tremendous advances are not without their chal-
lenges. One of the most pressing concerns is the vulnerability
of perception systems to adversarial attacks. Adversarial at-
tacks involve deliberately manipulating input data to deceive or
confuse perception systems, leading to potentially dangerous
outcomes on the road.
These attacks can take various forms, such as adding
imperceptible perturbations to images or videos, strategically
placing objects in the environment, or exploiting the algo-
rithmic behavior of perception systems. The consequences of
successful adversarial attacks on perception systems can be
severe, ranging from misclassification of objects to complete
system failure.
Adversarial attacks threaten the safety and reliability of self-
driving cars, undermining the trust and confidence of both
passengers and regulators in autonomous vehicle technology.
Addressing this challenge requires robust defense mechanisms
and rigorous testing protocols to detect and mitigate potential
vulnerabilities in perception systems.
Significant advancements in perception algorithms have
spurred extensive research into adversarial analysis within
vehicular perception systems in recent years. However, this
body of research lacks organization, making it challenging
for scholars and practitioners to navigate when exploring the
literature on this subject. The existing literature on adversarial
analysis in perception may be scattered, hindering efficient
retrieval and synthesis of relevant information. This lack of
cohesion and centralization in the research landscape poses a
significant obstacle to the advancement of knowledge in this
domain. Our work bridges those gaps and provides detailed
insights into existing research. Furthermore, we explore and
provide suggestions for future research directions toward im-
proved mitigation strategies and possible attack vectors that
require further exploration.
The contribution of the paper are as follows:
 • We provide a systematic organization of the knowledge
around vehicular perception systems, categorizing known
adversarial attacks into three broad domains: sensor-
based, model-based, and miscellaneous attacks.
• We provide a detailed evaluative study on feasibility,
stealthiness, transferability, and robustness of these at-
tacks.
• We finally provide a detailed outlook on potential future
research directions exploring new attack vectors and
improving defenses for vehicular perception.
II. R ELATED W ORKS
In this section, we provide a review of previous studies
conducted on adversarial attacks in perception systems. The
seminal work by Szegedy et al. [] introduced the concept
of adversarial examples, demonstrating that imperceptible
perturbations could cause deep neural networks (DNNs) to
misclassify inputs. Since then, researchers have extensively
studied adversarial attacks across various perception systems,
including image classification, object detection, and natural
language processing.
III. S ENSOR - BASED P ERCEPTION ATTACKS
In this section, we categorize attacks primarily centered
on introducing errors into sensor readings as these sensors
gather data from the environment. Given the immense scope of
sensor attacks, we undertake the substantial task of organizing
them into broad domains. The categorization aids readers in
navigating through attacks targeting specific types of sensors.
A. LiDAR
LiDAR sensors are one of the expensive sensors that
are on-board an autonomous vehicle providing a detailed
environments information around them. LiDARs provide a
3D overview of the surrounding by projecting laser pulses
periodically and observing the time taken to receive the laser
signals. Compared to the information captured by a camera
sensor, LiDARs provide accurate depth information about their
surrounding. However, LiDAR-based perception are vulnera-
ble to spoofing attacks. The attacker spoof the coordinates of
an imaginary vehicle in front of the victim autonomous vehicle
through the utilization of strategically timed laser signals to the
victim’s LiDAR sensor [1]. The spoofed signals take advantage
of the ignored occlusion patterns by the object detection model
in cars.
B. Camera
Cameras are crucial in providing accurate environmental
perception to an autonomous driving system (ADS) by aiding
in detection of various objects in a scene, such as lanes,
road signs, and pedestrians. Furthermore, cameras precisely
simulate human vision, helping in enhancing system’s adapt-
ability to different environments by observing road textures,
lighting variations, and weather condition. However, despite
their advantages, cameras are susceptible to physical attacks
which can compromise the safety of the autonomous vehicle.
2
For example, road signs can be covered or manipulated, or
additional lines can be painted to confuse the lane detection
system [2]. Based on the existing literature, physical camera
attacks can be broadly categorized into two, proximal attacks,
where the attacker is in close range to the target vehicle it,
while the vehicle may be moving or stationary, and stationary
vector attacks where the attacker is stationary and road objects
are manipulated to confuse the perception layer. such as
tampering with road signs or road lanes.
1) Blinding Attack: Blinding attacks are a form of proximal
attacks where an attacker utilizes light emitting devices such
as lasers in close proximity of the target vehicle. This form
of attack exploits auto-adjustment done by the camera to the
varying light intensities, a key vulnerability in any camera. The
disruption with the blinding light can hinder the perception
about oncoming traffic that can convert to a life-threatening
outcome. Petit et al. [2] illustrated this in their experiments
with numerous varying light sources which were activated at
varying distances, resulting in momentary blindness to the
camera sensor C2-270, a camera specifically designed for
mitigating collisions.
C. RADAR
Radar systems are crucial for providing accurate spatial
awareness to an autonomous driving system (ADS), aiding in
the detection of obstacles like other vehicles and pedestrians.
They are particularly valuable for their ability to operate
effectively under various environmental conditions, including
poor visibility. However, radar systems are not immune to
manipulation and can be compromised through specific types
of attacks. For instance, attackers can employ proximal radar
jamming or spoofing tactics. Proximal jamming involves using
devices that emit radio waves matching the frequencies used
by the radar, overwhelming it with false signals and disrupting
its ability to accurately measure distances and velocities.
Spoofing, on the other hand, involves the creation of fake
radar signals that trick the system into perceiving non-existent
objects or incorrectly positioning actual ones. These attacks
can severely impair the ADS’s functionality, leading to unsafe
driving decisions. The existing literature categorizes these as
either proximal attacks, where the attacker operates close to
the moving or stationary vehicle, or more distanced attacks
where the interference is indirect yet specifically targets the
radar’s functionality to degrade system performance.
D. GPS
GPS systems are vital for navigation in autonomous driving
systems (ADS), offering precise location tracking to facilitate
route planning and vehicle positioning. However, GPS is
susceptible to certain types of attacks that can undermine
its accuracy and reliability. These include GPS spoofing and
signal jamming. In GPS spoofing, an attacker transmits fake
GPS signals that the vehicle’s receiver mistakes for authentic
satellite signals, leading the ADS to calculate an incorrect po-
sition. This can misdirect the vehicle to unintended locations or
disrupt its geographical awareness. GPS jamming, conversely,
involves the use of devices that emit radio noise or signals at
the same frequency as GPS satellites, effectively drowning out
the real signals and causing the receiver to lose track of its
location. These disruptions can compromise navigation safety,
leading to potential accidents or deviations from intended
paths. The literature classifies these threats into two broad
categories: proximal attacks, where the attacker is in close
proximity to the target vehicle, whether moving or stationary,
and more sophisticated, distanced attacks that manipulate the
GPS signal itself, misleading the ADS’s perception of its
environment.
IV. M ODEL - BASED P ERCEPTION ATTACKS
In this section, we systematically organize a comprehensive
review of research dedicated to adversarially attacking the
perception systems of autonomous vehicles (AVs). We exam-
ine a spectrum of adversarial tactics, ranging from intricately
crafted examples tailored to exploit vulnerabilities in white-
box models to robustly effective attacks capable of deceiving
multiple black-box perception models. We meticulously cat-
egorize these adversarial attacks based on the particular per-
ceptions they aim to compromise. The following categorization
provides a lucid and structured exposition of our analysis.
A. LiDAR-based Perception
Model-based perception attacks on LiDAR systems are a
sophisticated threat in autonomous driving systems (ADS),
targeting the core algorithms that interpret environmental
data. These attacks manipulate the LiDAR input, causing the
perception models to misidentify or overlook objects, thereby
compromising navigational decisions. For instance, attackers
might inject false objects into the LiDAR’s point cloud or alter
the representation of actual objects. This results in the ADS
either evading non-existent obstacles or ignoring real hazards,
posing severe safety risks.
The impact of such attacks is abject, potentially leading to
accidents or erroneous vehicle behaviour in traffic. Defend-
ing against these threats involves enhancing the algorithms’
resilience to abnormal inputs, employing comprehensive val-
idation of sensor data, and integrating redundancy across
multiple sensing technologies to verify data consistency. These
measures are crucial for maintaining the integrity and safety
of ADS operations in the face of increasingly sophisticated cy-
ber threats.
B. Camera-based Perception
Camera-based attacks on autonomous driving systems
(ADS) pose a significant threat by targeting the cameras that
are crucial for the vehicle’s ability to see and understand its
surroundings. These attacks typically involve altering visual
elements within the vehicle’s environment, such as distorting
or covering road signs, or creating fake lane markers. For
example, attackers might place stickers or paint on a stop sign
to make it appear as a yield sign, or project images onto the
road to simulate non-existent obstacles. This can cause the
vehicle to make dangerous decisions, such as ignoring real
3
stop signals or swerving to avoid imaginary objects, potentially
leading to traffic accidents or major disruptions.
To counter such threats, it’s essential to develop camera
systems with advanced image processing capabilities that
can identify and ignore these deceptive inputs. Furthermore,
combining multiple types of sensors, like radar and LiDAR,
with cameras helps verify the information each sensor pro-
vides, enhancing the overall reliability and safety of the ADS.
Implementing such multi-layered defense strategies is crucial
to protect autonomous vehicles from these visual perception
attacks and ensure they can navigate safely through complex
environments.
1) Adversarial Attacks: Start with brief description of what
is an adversarial attack Adversarial attacks intend to mislead
the classification results by introducing imperceptible pertur-
bations to mislead the models in producing wrong results
with significant confidence. Adversarial attacks can be broadly
classified into three categories, white-box, black-box, and
grey-box attacks. In white-box attacks, the adversary has
complete knowledge of the target model such as its parameters,
the loss function, and the dataset involved in training the
model. Black-box attacks are done by an adversary, when the
model is available, however, anything else about the model
is not known. The adversary can only infer based on the
inputs and outputs from the target model. In contrast, grey-box
attacks involve partial awareness about the target model which
can significantly lower the barrier to interpret the model’s
mechanism.
Within these attack frameworks, numerous attacks algo-
rithms have been introduced for adversarial sample genera-
tions. The paper further discusses how we can defend against
these attacks and what is the effectiveness and efficiency of
these defenses, which will be discussed in sectionwrite the
section number where you have discussed/will be discussing.
C. RADAR-based Perception
Radar-based attacks on autonomous driving systems (ADS)
are significant concerns due to their potential to disrupt the
radar sensors that help vehicles detect and respond to objects
in their path. These attacks might involve jamming the radar
with competing signals or spoofing, where false signals are
sent to trick the radar into seeing objects that aren’t there. For
instance, an attacker could use a jamming device to flood the
radar with noise, making it difficult for the system to discern
real objects like cars or pedestrians. Alternatively, spoofing
could make the radar perceive phantom obstacles, leading
the vehicle to make unnecessary maneuvers, such as abrupt
braking or swerving, creating unsafe driving conditions.
To safeguard against these types of threats, it’s critical to
equip radar systems with filters and algorithms designed to
distinguish between legitimate and fraudulent signals. Ad-
ditionally, incorporating a fusion of different sensor tech-
nologies, like cameras and LiDAR, alongside radar can help
cross-validate data, enhancing detection accuracy and system
resilience. This multi-sensor approach is essential in building
robust defenses for ADS, ensuring they can operate safely and
effectively even in the presence of potential radar interference.
V. M ISCELLANEOUS ATTACKS ON P ERCEPTION S YSTEMS
This section aims to categorize attacks that were not intently
focused on one particular kind of perception but can severely
affect the navigation of an AV significantly. These adversar-
ial attacks are mainly focused on challenging fundamental
concepts of perception that fuel the autonomous navigation
system. It can range from novel perspectives to downstreaming
tasks that follow after the perception algorithms have identified
the region of interests in the scene.
A. Motion-planning/Trajectory Planning
B. End-to-End Driving Stack
C. Collaborative Perception on Connected AV
VI. D EFENSES AGAINST P ERCEPTION ATTACKS
A. Defense against adversarial model-based attacks
1) Adversarial Training: In this section we will be dis-
cussing about the importance and various methods of im-
plementing adversarial training for a deep neural network
and machine learning models. To increase the robustness and
efficiency of models against adversarial attacks, adversarial
training methods are introduced during machine learning stage.
Perturbed adversarial samples can deceive the model leading
to inaccurate predictions, however, gradually introducing them
during the training stage creates robustness in the model and
it is able to make better predictions. we will further discuss
the samples and defense mechanisms against those samples.
Over the recent years, numerous methods have been dis-
cussed to improve defenses against adversarial attacks. Adver-
sarial training being the first and foremost, where perturbed
adversarial samples are introduced during the training stage,
which allows the model to effectively identify these adversarial
samples and improve the accuracy of prediction. Numerous al-
gorithms and methods [3] have been researched for generating
these samples to introduce into the training stage.
Fast gradient sign method (FGSM) [4] is an efficient white-
box attack which introduces noise rendering the model in
predicting the wrong results with high confidence.Basic Iter-
ative Method (BIM) and Projected gradient descent(PGD) [3]
algorithms implement taking steps in the negative directions of
the gradient to train the model with perturbed data resulting in
wrong predictions within feasible constraints. After elaborate
testing and training of model on the MNIST and CIFAR-10
databases [5]. Furthermore, to avoid computational constraints,
the author [3] provides detailed outlook on ensemble adver-
sarial training. All of these adversarial training algorithms
were tested against large number of datasets such as MNIST,
CIFAR-10 and ImageNet, to test their robustness, efficiency,
and effectiveness.
Another method discussed to implement defenses against
adversarial attacks is to introduce randomization schemes [6].
Such as, random input transformations applied to input images
increased robustness of the CNN’s. various methds discussed
4
were resizing or adding padding to the images, these methods
were tested with a subset of images from the ImageNet dataset
Fast Gradient Sign Method (FGSM): This method involves
calculating the gradient of the loss function with respect to the
input data to create adversarial examples. Projected Gradient
Descent (PGD): This method enhances resistance against a
wide range of first-order attacks. Ensemble Adversarial Train-
ing: To mitigate computational constraints, this method trains
models on a combination of adversarial examples generated
from multiple models. These algorithms were tested on various
datasets to evaluate their robustness, efficiency, and effective-
ness.
Additionally, the paper discusses randomization schemes
as another defense mechanism. For example, random input
transformations applied to images increased the robustness
of CNNs, resisting 60% of strong grey-box attacks and 90%
of strong black-box attacks. Other techniques include random
noising and random feature pruning.
B. Defense against sensor based attacks
1) Redundancy: Redundancy is a crucial factor in enhanc-
ing the reliability and security of autonomous driving (AD)
systems. Extensive physical tests were conducted, involving
the placement of lasers with varying intensities and at different
distances from a camera. The results indicated that the lasers
could effectively blind the camera by preventing it from tuning
its exposure, which led to overexposed images. The main
parameters varied during these tests were the distance of the
laser from the camera and the intensity of the laser.
To successfully defend against such real-world attacks,
Petit et al., [2] proposed a solution involving redundancy in
the sensor layer was proposed. It was observed that placing
multiple cameras on the AD vehicle, at different observa-
tion angles and distances, significantly increased the effort
required by an attacker to blind all cameras simultaneously.
For instance, positioning cameras on the front bumper and
windshield introduced varying perspectives that made it more
difficult for a single laser attack to compromise the system.
The data from these multiple cameras is later combined to
form a single cohesive image. However, aligning the cameras
correctly is essential to ensure the images overlap properly,
which, though a trivial task, presents challenges. One of the
primary challenges is capturing synchronized images with
the same exposure settings to maintain the integrity of the
merged image. This redundancy strategy not only enhances the
robustness of AD systems but also significantly mitigates the
risk of successful attacks, thereby improving overall safety.
VII. D IRECTIONS U NDEREXPLORED
R EFERENCES
[1] J. Sun, Y. Cao, Q. A. Chen, and Z. M. Mao, “Towards robust {LiDAR-
based} perception in autonomous driving: General black-box adversarial
sensor attack and countermeasures,” in 29th USENIX Security Symposium
(USENIX Security 20), 2020, pp. 877–894.
[2] J. Petit, B. Stottelaar, M. Feiri, and F. Kargl, “Remote attacks on
automated vehicles sensors: Experiments on camera and lidar,” Black Hat
Europe, vol. 11, no. 2015, p. 995, 2015.
[3] K. Ren, T. Zheng, Z. Qin, and X. Liu, “Adversarial attacks and defenses
in deep learning,” Engineering, vol. 6, no. 3, pp. 346–360, 2020.
[4] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing
adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
[5] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards
deep learning models resistant to adversarial attacks,” arXiv preprint
arXiv:1706.06083, 2017.
[6] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. Yuille, “Mitigating adversarial
effects through randomization,” arXiv preprint arXiv:1711.01991, 2017.