Self Driving AI Cars

Akshay Karthik Anant Vikram Singh

Dept. of Computer Science and Engineering Dept. of Computer Science and Engineering
RNS Institute of Technology RNS Institute of Technology
Bangalore, India Bangalore, India
1rn19cs012.akshay@gmail.com 1rn19cs015.anantvikramsingh@gmail.com

Anubhab Saha Astitva Singh

Dept. of Computer Science and Engineering Dept. of Computer Science and Engineering
RNS Institute of Technology RNS Institute of Technology
Bangalore, India Bangalore, India
1rn19cs016.anubhabsaha@gmail.com 1rn19cs022.astitva@gmail.com

Dr. Kiran P
Head of Department
Dept. of Computer Science and Engineering
RNS Institute of Technology
Bangalore, India

Abstract—Self-driving cars are an emerging technology that
utilizes advanced machine learning algorithms, including deep
learning, to enable vehicles to navigate roads and traffic without
human intervention. This technology has the potential to revo-
lutionize the transportation industry by improving road safety,
reducing congestion, and increasing mobility for people who
cannot drive or do not have access to transportation. Deep
learning is a subset of artificial intelligence which contains
algorithms to train computers using copious amounts of data, this
is done so that it doesn’t have to be programmed individually.
These algorithms are critical to self-driving cars as they allow
the vehicle’s sensors, such as cameras and lidar, to process and
analyze visual data from the surrounding environment to make
informed decisions. One of the primary challenges of developing
self-driving cars is ensuring their safety in unpredictable and
complex driving scenarios. Deep learning algorithms enable
vehicles to learn from a vast amount of data, including diverse
and rare driving scenarios, to improve their decision-making
ability. Deep learning architectures, such as Convolutional neural
networks(CNNs) and Recurrent neural networks(RNNs), have
been used to train the AI of self-driving cars. Convolutional
neural networks have been used to process visual data from cam-
eras, Lidars and Radars to identify objects, such as jay-walking
pedestrians, traffic lights, road blockages and other vehicles.
Recurrent neural networks have been used to understand and
predict the movement of other vehicles, pedestrians and animals
to anticipate potential collisions. The development of self-driving
cars using deep learning algorithms is a rapidly evolving field.
As the technology continues to advance, it has the potential to
transform the transportation industry, improve road safety, and
increase access to transportation for people around the world.
Index Terms—Convolutional Neural Networks, Recurrent neu-
ral networks, lidar, radar, deep learning, self driving
I. I NTRODUCTION
Self-driving AI cars, also known as autonomous vehicles
(AVs), are a rapidly developing technology that promises to
revolutionize the way we travel. These vehicles use a combina-
tion of sensors, cameras, and advanced algorithms to interpret
the surrounding environment and make decisions about how to
safely navigate the road. The development of self-driving AI
cars is being driven by several factors, including the desire to
reduce traffic congestion, increase safety, and improve access
to transportation for people who cannot drive. In this article,
we will explore the key features and benefits of self-driving
AI cars and the challenges that must be overcome to make
them a reality.
A. Key Features of Self-Driving AI Cars
Self-driving AI cars are equipped with a variety of sensors
and cameras that allow them to perceive the world around
them. These sensors include :
1) Lidar: A laser-based sensor that creates a 3D map of the
car’s surroundings.
2) Radar: A radio-based sensor that detects the presence
and distance of objects.
3) Cameras: High-resolution cameras that capture visual
information about the environment.
4) GPS: A global positioning system that provides location
information.
5) Inertial measurement unit (IMU): A sensor that mea-
sures the car’s acceleration and rotation.
These sensors work together to create a detailed map of
the car’s surroundings, which is then processed by advanced
algorithms to make decisions about how to navigate the road.
B. Benefits of Self-Driving AI Cars
Self-driving AI cars offer a number of benefits, including:
 1) Improved safety: Self-driving AI cars are designed to
reduce the number of accidents caused by human er-
ror. According to the National Highway Traffic Safety
Administration, 94% of accidents are caused by human
error, and self-driving AI cars have the potential to
significantly reduce this number.
2) Increased mobility: Self-driving AI cars can provide
transportation for people who are unable to drive, such
as the elderly or people with disabilities. This can
help to improve their quality of life and increase their
independence.
3) Reduced traffic congestion: Self-driving AI cars are ex-
pected to reduce traffic congestion by optimizing traffic
flow and reducing the number of cars on the road.
4) Improved efficiency: Self-driving AI cars can be pro-
grammed to drive in a more fuel-efficient manner, reduc-
ing carbon emissions and improving the environment.
C. Challenges of Self-Driving AI Cars
While self-driving AI cars offer many benefits, there are
also a number of challenges that must be overcome before
they can become a reality. These challenges include:
1) Safety: Self-driving AI cars must be able to operate
safely in all conditions, including bad weather, heavy
traffic, and unforeseen events such as accidents or road
closures. Ensuring the safety of self-driving AI cars is a
major challenge that must be addressed before they can
become widely adopted.
2) Regulation: The development and deployment of self-
driving AI cars are subject to a complex web of regula-
tions and laws that must be navigated. These regulations
vary by country and state and must be carefully followed
to ensure that self-driving AI cars can be legally oper-
ated.
3) Infrastructure: Self-driving AI cars require a complex
infrastructure to operate, including advanced sensors and
communication networks. Building this infrastructure is
a major challenge that must be addressed before self-
driving AI cars can become a reality.
4) Public acceptance: Self-driving AI cars are a new tech-
nology, and many people may be hesitant to trust them.
Ensuring that the public is comfortable with self-driving
AI cars is a major challenge that must be addressed to
ensure their widespread adoption.
II. L ITERATURE S URVEY
Self-driving AI cars are one of the most fascinating
advancements in the field of artificial intelligence and
transportation. Self-driving cars have been an active area of
research and development for several years. In this literature
survey, we will explore the different aspects of self-driving
AI cars, including their history, development, and prospects.
1) History of Self-Driving AI Cars: The idea of self-
driving cars has been around for decades, with the first
prototypes developed in the 1980s. However, it was not
until the 2000s that significant progress was made in
this area, with the development of advanced sensors,
machine learning algorithms, and other technologies.
In 2004, the DARPA Grand Challenge was launched,
which encouraged researchers and engineers to
develop self-driving cars that could navigate through
difficult terrain and complete complex tasks without
human intervention. This challenge sparked a wave
of innovation and investment in the field, with major
companies like Google, Uber, and Tesla investing
heavily in self-driving AI technology.
2) History of Self-Driving AI Cars: The idea of self-
driving cars has been around for decades, with the first
prototypes developed in the 1980s. However, it was not
until the 2000s that significant progress was made in
this area, with the development of advanced sensors,
machine learning algorithms, and other technologies.
In 2004, the DARPA Grand Challenge was launched,
which encouraged researchers and engineers to
develop self-driving cars that could navigate through
difficult terrain and complete complex tasks without
human intervention. This challenge sparked a wave
of innovation and investment in the field, with major
companies like Google, Uber, and Tesla investing
heavily in self-driving AI technology.
3) Challenges and Opportunities: Despite the significant
progress made in self-driving AI cars, there are
still many challenges and opportunities facing this
technology. One of the biggest challenges is ensuring
the safety and reliability of self-driving cars, especially
in complex and unpredictable environments. Other
challenges include the legal and regulatory frameworks
needed to govern self-driving cars, as well as the ethical
and social implications of this technology. However,
there are also many opportunities associated with
self-driving AI cars.
4) Recent Research and Publications: There has been a
significant amount of research and publications in the
field of self-driving AI cars in recent years. One re-
cent study by researchers at the Massachusetts Institute
of Technology (MIT) examined the ethical and moral
dilemmas associated with self-driving cars and proposed
a framework for making ethical decisions in autonomous
driving situations. Another study by researchers at Stan-
ford University explored the use of machine learning
algorithms to improve the safety and reliability of self-
driving cars, by predicting potential accidents and taking
preemptive measures to avoid them.
III. A RCHITECTURE
Some of the algorithms used in the aforementioned project
are, YOLO and Faster R-CNN for object detection tasks.
YOLO short for “You Only Look Once” more precisely
YOLO version-8, is a real-time object detection algorithm that
provides high-accuracy outputs by dividing the image into a
grid and predicting bounding boxes, and classifying the object
bounded by the box into one of the names described by the
programmer during training along with the probability of the
classification. On the other hand, Faster R-CNN is a region-
based Convolutional neural network that proposes candidate
regions in an image and applies a classifier to each proposed
region to identify objects. Even though Faster R-CNN had
higher accuracy and robustness when compared to YOLO v8
in the detection of smaller objects it was spotted to be dilatory
and unpractical for this particular application, which needed
instantaneous, real-time image processing. Hence, YOLO v8
was found to be more suitable.
A. Yolo
YOLO v8 is a state-of-the-art object detection algorithm that
achieves high accuracy and real-time performance by utilizing
a unique architecture that combines a backbone network,
neck network, and head network. The backbone network,
which is based on the DarkNet architecture, is responsible for
extracting features from the input image. This network consists
of 13 convolutional layers followed by 3 max-pooling layers.
The neck network, also known as the SPP (Spatial Pyramid
Pooling) module, is responsible for aggregating features from
multiple scales. This network consists of four pooling layers,
each with a different output size. The outputs of the pooling
layers are concatenated and fed into several fully connected
layers to produce a fixed-length feature vector. Finally, the
head network is responsible for predicting object classes and
bounding boxes. This network consists of several convolu-
tional layers followed by two fully connected layers. The
output of the head network is a tensor with a fixed shape,
which is then processed to produce bounding boxes and class
probabilities. To better understand the YOLO v8 architecture,
we can look at its block diagram. The input image is first fed
into the backbone network, which consists of convolutional
layers and max- pooling layers. The output of the backbone
network is then passed through the neck network, which
consists of four pooling layers and fully connected layers. The
output of the neck network is a fixed-length feature vector
that is then fed into the head network. The head network
consists of several convolutional layers and fully connected
layers and produces a tensor with a fixed shape that represents
the predicted object classes and bounding boxes. The output
tensor is then processed to produce final object detection
results, which include the object classes, bounding boxes, and
confidence scores. Overall, the YOLO v8 algorithm is a highly
effective and efficient object detection algorithm that utilizes a
unique architecture to achieve real-time performance with high
accuracy. The below figure, Fig 1, shows the block diagram
of YOLO v8 architecture used to build the Self Drive AI Car.
Fig. 1. Detailed Block Diagram of YOLO Architecture (Source: Research-
Gate)
B. Faster R CNN:
Faster R-CNN is a popular object detection algorithm that is
widely used in computer vision applications. It was proposed
by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun in
2015. The algorithm is built on the basis of R-CNN and Fast
R-CNN, which are the earlier versions of the algorithm. Faster
R-CNN is designed to overcome the limitations of the previous
versions, such as the slow processing speed and the need for
multiple stages of computation. The Faster R-CNN algorithm
consists of two main parts: a region proposal network (RPN)
and a region-based convolutional neural network (R-CNN).
The RPN is responsible for generating region proposals that
are likely to contain objects. The R-CNN uses these proposals
to classify the objects and predict their bounding boxes. The
RPN is a fully convolutional network that takes an image
as input and outputs a set of object proposals, which are
rectangular regions of interest (RoIs) that are likely to contain
objects. The RPN is trained to maximize the overlap between
the predicted proposals and the ground- truth bounding boxes
of objects. The R-CNN then takes these RoIs as input and
extracts a fixed-length feature vector for each of them using
a convolutional neural network. The feature vectors are then
fed into a set of fully connected layers that classify the
objects and predict their bounding boxes. Faster R-CNN uses
a shared convolutional neural network to extract features from
the input image. This network is trained on a large dataset,
such as ImageNet, to learn features that are useful for object
detection. The RPN and the R-CNN share the same network
up to the feature extraction layer, which allows them to
share the computation and reduce the processing time. Faster
R-CNN has several advantages over its predecessors. It is
faster than R-CNN and more accurate than Fast R-CNN. It
also has a simpler architecture than its predecessors, which
makes it easier to implement and train. Additionally, Faster
R-CNN can detect objects of different scales and aspect ratios, V. R ESULT A NALYSIS
which makes it more robust to variations in object size and
shape. Overall, Faster R-CNN is a powerful algorithm that has
become a standard in the field of object detection.
The below figure, Fig 2, shows the block diagram of Faster
R-CNN architecture used to build the Self Drive AI Car.

Fig. 3. A view from the object detections in a traffic video. Objects are
labeled correspondingly.

Fig. 2. Block Diagram of Faster R-CNN (Source: ResearchGate)

IV. S TAGES OF THE P ROJECT

The process of training an object detection model using
a time-lapse video of the college started with collecting the
video data and labeling the objects of interest. The video
frames were analyzed to identify the objects such as vehicles,
pedestrians, and bicycles. The annotated data was used to train
the Faster R-CNN algorithm. The training process involved
optimizing the neural network’s weights to minimize detection
errors. After the training process, the model was tested on live
cameras on the college campus. However, there was a delay
Fig. 4. Table of all the labels used from the COCO dataset
in the detection due to the use of Faster R-CNN. Therefore,
it was decided to switch to a faster object detection algorithm
called YOLO v8 (You Only Look Once version 8). YOLO
v8 is a real-time object detection algorithm that performs
object detection in a single pass, making it much faster than
Faster R-CNN. To further improve the model’s performance,
an ultrasonic-based object detection and avoidance algorithm
was incorporated into the system. This algorithm helps the
model avoid obstacles in the environment and navigate around
them to avoid collisions. The testing of the model was done
in a simulator made in the Unreal Engine. The simulator
allowed for the model to be tested in a virtual environment
that closely mimics the real world. This testing allowed for the
fine-tuning of the model’s performance and the identification
of any remaining issues that needed to be addressed. Once the
model was fully tested and optimized, it was integrated into Fig. 5. Continuous live output of the objects identified in the image.
an RC car using an Arduino Uno and a stepper motor. The RC
car was equipped with 5 ultrasonic sensors and the data from
that was used to detect objects in the environment in real time.
The ultrasonic-based object avoidance algorithm was used to
guide the car around obstacles in its path.
 VI. C ONCLUSION
In conclusion, the development of self-driving cars has
the potential to revolutionize the way we travel and signif-
icantly improve road safety. While there are still technical
and regulatory challenges to be addressed, the progress made
in recent years is promising. Self-driving cars can provide
benefits such as reduced traffic congestion, increased mobility
for the elderly and disabled, and improved energy efficiency.
However, there are also potential drawbacks, such as job losses
in the transportation sector and ethical dilemmas related to
decision-making algorithms. As we continue to advance in
this field, it is important to carefully consider the implications
and ensure that the technology is developed in a responsible
and ethical manner. Overall, self-driving cars offer exciting
possibilities for the future of transportation, and it will be
fascinating to see how this technology develops and impacts
our society.
ACKNOWLEDGMENT
People are continually appreciative of others for their aid
and direction in achieving their goals. This formal letter of
thanks is an attempt to express our gratitude to everyone who
assisted us in realizing this project and kept us motivated
despite the challenges it presented.
R EFERENCES
[1] ”A Survey on Autonomous Vehicles” by Ali Al-Maashri and Salah Al-
Majeeni, in IEEE Access, 2019.
[2] ”Autonomous Cars: Past, Present and Future - A Review of the Develop-
ments in the Last Century, the Present Scenario and the Expected Future
of Autonomous Vehicle Technology” by Shukla et al., in International
Journal of Engineering Research and Applications, 2019.
[3] ”Artificial Intelligence for Self-Driving Cars: A Review” by Muhammad
Arif and Sajjad Ahmed, in International Journal of Advanced Computer
Science and Applications, 2021.
[4] ”Deep Reinforcement Learning for Autonomous Driving: A Survey”
by Li and Liang, in IEEE Transactions on Intelligent Transportation
Systems, 2020.
[5] ”A Survey on Sensor Fusion in Autonomous Vehicles” by Mohan and
Verma, in Sensors, 2020.
[6] ”Machine Learning Techniques for Autonomous Driving: A Survey”
by Sermanet and LeCun, in IEEE Intelligent Transportation Systems
Magazine, 2018.
[7] ”A Survey on Motion Planning and Control Techniques for Autonomous
Driving” by Bödefeld et al., in IEEE Transactions on Intelligent Trans-
portation Systems, 2020.
[8] ”A Comprehensive Review on Deep Learning Techniques for Au-
tonomous Vehicles” by Vignesh and Karthik, in Journal of Ambient
Intelligence and Humanized Computing, 2021.
[9] ”A Survey of Autonomous Vehicle Simulators” by Katzourakis and
Khan, in Robotics and Autonomous Systems, 2020.
[10] ”A Review of Deep Learning for Autonomous Driving” by Tamaazousti
et al., in Neural Computing and Applications, 2021.