SELF-LEARNING CAR SIMULATION USING NEAT

Krishna Ghoman1, Binju Chenthitta2, Prasad Sonawane 3, Vaibhav Kodag4, Prof. S.B. Akhade5
1, 2, 3, 4 Computer Department K J College Of Engineering and Management Research, Pune, India
5 Professor,
K J College Of Engineering And
Management Research Pune, India

Abstract: Self-driving car development has been a subject of study for many years. The way we commute could be completely
changed by autonomous vehicles, which could also help with traffic congestion and increase road safety. Self-driving cars have
recently benefited from the application of machine learning techniques to enhance performance. This project report showcases a
self-learning automobile simulation (NeuroEvolution of Augmenting Topologies). Using machine learning, NEAT evolves
neural networks to carry out a particular task. The simulation of a self-driving automobile is made to learn from mistakes, adapt
to new situations, and enhance its performance over time. To reach this goal automakers have started working in this area to
realize the potential and solve the challenges currently in this area to reach the expected outcome.
Index : NEAT , Self-driving.

I. INTRODUCTION

Self-driving car development has been a subject of study for many years. The way we commute could be completely changed by
autonomous vehicles, which could also help with traffic congestion and increase road safety. NEAT is a machine learning technique
that evolves neural networks to perform a specific task. The self-driving car simulation is designed to learn from its experiences,
adapt to new environments, and improve its performance over time. The simulation environment is created using Unity, a popular
game engine, and the car is equipped with sensors that detect obstacles and other cars on the road. The NEAT algorithm evolves the
neural network's structure and weights to optimize the car's performance. Breaking it down, the process of Reinforcement Learning
involves these simple steps:

 Observation of the environment

 Deciding how to act using some strategy
 Acting accordingly
 Receiving a reward or penalty
 Learning from the experiences and refining our strategy
 Iterate until an optional strategy is found

Let’s now understand Reinforcement Learning by actually developing an agent to learn drive automatically on its own. We aim to
train a rough autonomous driving agent using either OpenAI gym or NEAT environment and explored the effects of various
environment configuration like reward functions on the agent training performance. Here it is a car which evades the traffic and
drops pedestrians and also potholes to reach the destinations.

Summary of WPF

You can develop Windows desktop client apps with visually attractive user interfaces using the Windows Presentation Foundation
(WPF) technology.
A vector-based, resolution-independent rendering engine that is built to make use of cutting-edge graphics hardware forms the basis
of WPF. Extensible Application Markup Language (XAML), controls, data binding, layout, 2D and 3D graphics, animation, styles,
templates, documents, media, text, and typography are all part of the extensive range of application-development tools that WPF
adds to the core. You can create apps that use other.NET API components because WPF is a component of.NET.
This review of WPF's main features and principles is written with beginners in mind.
Script using WPF
A subset of.NET types, known as WPF, are present and are mostly found in the System.windows namespace. The basic WPF
programming experience should be familiar if you have previously created applications with the.NET framework using managed
technologies like ASP.NET and Windows Forms; you instantiate classes, set properties, call methods, and handle events using your
preferred.NET programming language, such as C# or Visual Basic.
Dependency properties and routed events are two additional programming elements in WPF that improve properties and events.
Code-behind and markup
Developers acquainted with ASP.NET should be comfortable with WPF's ability to create applications utilising both markup and
code-behind. The behaviour of an application is typically implemented using managed programming languages (code-behind),
whereas the look of the application is typically implemented using XAML markup. The following advantages result from this
distinction between look and behaviour:

 Development is more effective because designers can implement an application's appearance concurrently with developers
who are implementing the application's behaviour.
 Development and maintenance costs are decreased because appearance-specific markup is not tightly coupled with
behaviour-specific code.
 Globalisation and localization for WPF applications is facilitated.

Markup
XAML is an XML-based markup language that implements an application's appearance declaratively. You typically use it to create
windows, dialog boxes, pages, and user controls, and to fill them with controls, shapes, and graphics.
The following example uses XAML to implement the appearance of a window that contains a single button:

SharpNEAT - Evolution of Neural Networks

Kenneth O. Stanley created the evolutionary algorithm known as NEAT.

SharpNEAT is a complete NEAT implementation written in C# that is designed to work with.NET [Core] 7.

SharpNEAT : What is it?

A neural network population will evolve as part of SharpNEAT's implementation of an Evolutionary Algorithm (EA) with the
specific aim of resolving a goal problem task (also referred to as the Objective function).

The EA searches for a neural network that "solves" a given problem task using the evolutionary mechanisms of mutation,
recombination, and selection. Each neural network is given a fitness score that indicates the calibre of the solution it represents.

391
 II. LITERATURE SURVEY

1] Paper Name: Evolving Neural Networks through Augmenting Topologies

Author: Kenneth O. Stanley and Risto Miikkulainen
An important question in neuro-evolution is how to gain an advantage from evolving neural network topologies along with weights.
They present a method, Neuro-evolution of Augmenting Topologies (NEAT), which outperforms the best fixed-topology method on
a challenging benchmark reinforcement learning task. They claim that the increased efficiency is due to employing a principled
method of crossover of different topologies, protecting structural innovation using speciation, and incrementally growing from
minimal structure. They test this claim through a series of ablation studies that demonstrate that each component is necessary to the
system as a whole and to each other. NEAT is also an important contribution to GAs because it shows how it is possible for
evolution to both optimized and complexify solutions simultaneously, offering the possibility of evolving increasingly complex
solutions over generations, and strengthening the analogy with biological evolution. [1]

2]Paper Name: Hybrid Self Attention NEAT (A novel evolutionary approach to improve the NEAT algorithm)
Author: Saman Khamesian and Hamed Malek
This article presents a “Hybrid Self-Attention NEAT” method to improve the original Neuro-evolution of Augmenting Topologies
(NEAT) algorithm in high dimensional inputs. Although the NEAT algorithm has shown a significant result in different challenging
tasks, as input representations are high dimensional, it cannot create a well-tuned network to select the most important parts of the
input. In addition, we improve its overall performance with the help of a hybrid method to evolve the final network weights. The
main conclusion is that Hybrid Self-Attention NEAT can eliminate the restriction of the original NEAT. The results indicate that in
with raw pixels input with a much lower number of parameters. [2]

3]Paper Name: - Neuro-evolution in Games: State of the Art and Open Challenges
Author: - Sebastian Risi and Julius Togelius
Survey: - This paper surveys research on applying neuro-evolution (NE) to games. In neuro-evolution, artificial neural networks are
trained through evolutionary algorithms, taking inspiration from the way biological brains evolved. They analyse the application of
NE in games along five different axes, which are the role NE is chosen to play in a game, the different types of neural networks
used, the way these networks are evolved, how the fitness is determined and what type of input the network receives. The article
also highlights important open research challenges in the field. [3]

4]Paper Name: - An Independent Study of Reinforcement Learning and Autonomous Driving

Author: - Hanzhi Yang
Survey: - In this report, they have studied Q-learning algorithm in reinforcement learning and its combination with deep learning
algorithm as deep Q-learning network. They conducted two experiments, taxi and carto pole, to apply these algorithms to solve real
problems. Furthermore, they studied the applications of deep reinforcement learning in autonomous driving and conducted an
experiment to examine the influence of Q-learning network parameters and agent reward functions on the reinforcement learning
performance. The results show that DRL is a potentially powerful tool to solve complicated real-world issues but still has a lot of
limitations. To solve these limitations, either advanced RL algorithms or a combination of model-based deep learning and
reinforcement learning is needed.[4]

392
 III. NOTABLE ALGORITHM USED FOR SELF LEARNING

 Agent –
An AI system is composed of an agent and its environment. The agents act in their environment. The environment may contain
other agents. An agent is anything that can be viewed as:
• Perceiving its environment through sensors
• Acting upon that environment through actuators
Every agent can perceive its own actions (but not always the effects) This project utilizes an agent i.e., A Car model which aims to
learn to drive around a given set of environments and evade challenges along its way.

 Environment-
An environment in artificial intelligence is the surrounding of the agent. The agent takes input from the environment through
sensors and delivers the output to the environment through actuators. There are several types of environments:

• Fully Observable vs Partially Observable

• Deterministic vs Stochastic
• Competitive vs Collaborative
• Single-agent vs multi-agent
• Static vs Dynamic
• Discrete vs Continuous
• Episodic vs Sequential
• Known vs Unknown
In our project, our environment contains a roadway, and a couple of challenges along the way to increase the difficulty of the AI to
learn to drive.

 Observation-
Observations are perceptions of the agent in an environment. There are many percussions of observations an agent can do during
RL. Once the agent understands and observes the environment around it, it manages to maximize the profit in every iteration while
evolving in the scenario of RL. In this case, in driving the environment is partially observable because what’s around the corner is
not known by the agent.

 Action-
Action selection in AI systems is a basic system in which the problem can be analysed by the AI machine to understand what it has
to do next to get closer to the solution of the problem. Since AI systems are very complex, it causes the problem of action selection
where more amount of time and computation is required to achieve a task and the agents that are responsible for that will have to
work through more data to get the job done. Actions are nothing but the work done by the Car (in out context) to complete laps
around the given arena and their aim being to maximize profit and benefit their evolution without failing much to reach there, with
every iteration helping them evolve. [6]

393
  Reward-
Rewards are common learning incentives for animals. A squirrel, for instance, develops intellectual abilities in its search for nuts. A
child, meanwhile, may get a chocolate for tidying their room or get scolded for bad behaviour. In this project system, the rewards
and punishments are calculated mathematically. This self-driving system could receive a -1 when the model hits a wall, and a +1 if
it safely passes another car. These signals allow the agent to evaluate its performance. The algorithm then learns through trial and
error to maximize the reward — and ultimately, complete the task in the most desirable manner.[6]
In the below DF0 diagram, when the program is run, the Car model starts the learning algorithm given in the above flowchart. The
car casts multiple rays (6) towards the white border walls of the black roadways and helps determine the location of the crash if the
car does hit the white walls. When destroyed, the agent knows the cause of the crash along with the accumulated points. This info
helps the agent to evolve in the NEAT algorithm.

IV. METHODOLOGY

The Design: The proposed procedures taken to implement the system are described below.
The Simulation: Cars have to navigate through a course without going outside the track. A car has five sensors covering a front
facing range of 90 degrees to measure the distance to obstacles in a given direction. The readings of these sensors serve as the input
of the car's neural network. The output of the Neural Network then determines the car’s current acceleration and steering force.
The Neural Network:
Training the Neural Network - The weights of the Neural Network are trained using an Evolutionary Algorithm known as the
NEAT.

394
 Fig1. Proposed System

V. Results / Output

Multiple car models traverse, finds paths, navigates through given path in order to reach the destination. If the cars collide with
against the borders of the roads or given obstacle parameters, the instance of the cars are destroyed. At start of every genome, the
process is repeated, spawning multiple cars per genome, with updated evolved genome data. After the cars learn how to navigate
through the obstacles, they eventually learn how to reach their destination.

395
The NEAT logo features the acronym "NEAT" in bold capital letters, with the letters "N" and "T" being slightly taller than the
letters "E" and "A". The letters are in a dark shade of blue, with a lighter blue gradient filling the inside of each letter. The gradient
starts with a slightly lighter shade of blue at the top and gradually gets darker towards the bottom.

The overall design of the logo is simple and clean, with a modern and professional feel. The use of a gradient adds a sense of depth
and dimensionality to the logo, making it visually interesting.

396
With this drag and drop interface, users can rotate roads and then place them in a virtual environment where the neural network can
learn to navigate around them. Overall, NEAT's drag and drop type interface is a user-friendly feature that can help
democratize machine learning and make it more accessible to a wider audience.

397
 NEAT is a highly customizable machine learning algorithm, and there are several parameters that can be adjusted to fine-
tune its performance for a specific application. Some of the most important parameters include:

 Population size: This parameter determines the number of neural networks that are evolved in each generation. A larger
population size can lead to better results, but it also requires more computational resources.

 Fitness function: The fitness function is used to evaluate the performance of each neural network. It can be customized to
prioritize different objectives, such as speed or accuracy, depending on the specific application.

 Mutation rates: Mutation rates determine how often mutations occur in the neural networks during evolution. Higher
mutation rates can lead to more diversity in the population, but can also result in slower convergence.

 Crossover rate: The crossover rate determines how often genetic material is exchanged between parent neural networks
during reproduction. A higher crossover rate can lead to faster convergence, but can also reduce diversity in the population.

398
 V. CONCLUSION

The project is aimed to determine the effect of AI algorithms in the Indian automobile industry. Through group and pair learning
activities, the team established an open dialogue and created a reliable AI system for neural networking. The project offers a cost-
effective solution to study, enhance and improve the conditions of the Indian A.I driving car scenario. Despite the challenges related
to security and safety, the team believes that many of these obstacles will be resolved as more autonomous solutions are developed
in the transportation industry. The project's findings are expected to inspire further research and contribute to the continued
development of AI solutions in the industry.

REFERENCES

1) Kenneth O. Stanley and Risto Miikkulainen “Evolving Neural Networks through Augmenting Topologies”, IEEE.org

2) Saman Khamesian and Hamed Malek “Hybrid Self Attention NEAT (A novel evolutionary approach to improve the NEAT
algorithm)”, arXiv.org

3) Sebastian Risi and Julius Togelius “Neuro-evolution in Games: State of the Art and Open Challenges “, arXiv.org

4) Hanzhi Yang “An Independent Study of Reinforcement Learning and Autonomous Driving”, arXiv.org

5) Python (programming language) “https://en.wikipedia.org/wiki/Python_(programming_language) “, Wikipedia

6) Reinforcement learning: How rewards create intelligent machines

“https://thenextweb.com/news/how-rewards-work-in-reinforcementlearning-deepmind#:~:text=In%20AI%20systems%2C
%20the%20rewards,agent%20t o%20evaluate%20its%20performance.”

399
7) About - PyGame wiki “https://www.pygame.org/wiki/about”

8) Welcome to NEAT-Python’s documentation “https://www.pygame.org/wiki/about”

9) openai/gym: A toolkit for developing and comparing reinforcement learning algorithms. (github.com)
“https://github.com/openai/gym

400