Multi-Agent Reinforcement Learning for Autonomous Driving
Tirth Patel
Lakehead University - Computer Science
Email: tpatel16@lakeheadu.ca
Student Id: 1143729
Abstract— With the advancement of self-driving cars in
recent years, the notion of multi-agent systems, as well as rein-
forcement learning, has emerged, building on the technologies
of self-driving cars. In terms of technological advancement, it
will be quite important.for the coming years. Many researchers
have stepped forward to perform diverse experiments by
constructing simulators that support a wide range of agent
behaviours, surroundings, tasks, situations, and other associated
elements in order to determine the best potential agent actions
in any dilemma-like circumstance. However, there are several
problems and drawbacks to these publications.in certain ways,
whereas high efficiency and benefits in others additional vari-
ables As a result, this study examines and contrasts a number
of well-known organizations. Simulations were created and are
widely used in the community. in order to create a variety of
multi-agent reinforcement learning (MARL) models designed
specifically for self-driving vehicles.
Index Terms— [keywords] MARL, Reinforcement Learning,
Simulation, Autonomous Driving
I. INTRODUCTION
Various groundbreaking studies in the subject of au-
tonomous driving have been conducted, utilising a range
of algorithms centred on teaching agents how to respond
in challenging situations by engaging with other agents via
various communication strategies. The existing simulators
discussed in this paper are Scalable Multi-Agent Reinforce-
ment Learning Training School (SMARTS), Multi-Agent
Connected Autonomous Driving (MACAD-Gym), Multi-
Agent Driving Simulator (MADRaS), and Behaviour Bench-
mark (BARK), all of which are based on Reinforcement
Learning (RL) and include a variety of agents, environments,
scenarios, algorithms, communications, and many more. As
a result, the focus of this research is on comparing the above-
mentioned factors across all four existing frameworks in
order to gain a clear picture of which simulator is ideal for
each of the given domains. Before getting into the details of
the comparison, this paper gives a quick rundown of all four
simulators, including their design, significant features, and
overall work with back-end algorithms. Finally, the study
summarises the comparison table and presents surprising
information before concluding.
II. METHODS
Before we begin our method discussion, we’ll go over a
quick rundown of all four simulators, including how they
function, their design, significant features, architecture and
many more. The main goal of the summaries is to provide a
general overview of the simulators that are already available
for better future Implementations.
A. SMARTS: Scalable Multi-Agent Reinforcement
Learning Training School for Autonomous Driving
SMARTS is an open-source, realistic, and multi-agent
simulator [6] that serves as a platform for the industrial
research and development of various Multi-Agent Reinforce-
ment Learning (MARL) based autonomous driving (AD),
allowing simulations to be extended for a real-world sce-
nario.SMARTS focuses on teaching agents so that they can
communicate with each other before concluding, just like
in the real world where genuine and diverse interactions
between different agents play a big role. SMARTS evaluates
aspects such as vehicle physics, actions of other road users,
the traffic scenario, the overall architecture and structure
of the environment and the roadways, as well as all the
rules and regulations to be obeyed in order to enable realis-
tic interactions. Aside from realistic interactions, SMARTS
is notable for having heterogeneous agents, primarily ego
agents (vehicle controlled by AD software), social agents
(vehicles sharing environment with autonomous vehicles),
and even hand-written social agents (i.e.agents following
only pre-defined specific tasks), trained using real-world data
and Reinforcement Learning (RL) algorithms, social agent
zoo which has a collection of varied agents that can be used
for making the model more robust as people can contribute
their agent algorithm design in the social zoo and thus, can
integrate and simulate AD among a variety of agents, making
SMARTS a great option for research [6]. It also has a feature
of visualizing the simulation in a web-based application,
along with recording the whole scenario [6].
In the SMARTS architecture from Fig. 1. the background
traffic provider is responsible for providing traffic scenarios
to which the agents must respond in terms of interactions
happening in the foreground. For the background traffic
provider, SMARTS uses Simulation for Urban Mobility
(SUMO) [2]. Moreover, for the interaction scenarios, it
uses Domain Specific Language (DSL) written in python,
which consists of pre-programming the SUMO and integrates
agents in the Social Zoo, with the routes, vehicles and overall
the map of the scenario in general. The bubble is a specific
region from the whole map where real interactions between
the environment and the vehicles take place. Furthermore,
the physics of the system mainly, ”throttle-brake-steering,
trajectory tracking, and lane following” (Zhou et. al., 2020) is
 Fig. 1. SMARTS Architecture
provided by the Vehicle Physics Provider. Lastly, the motion
plan provider is used for executing social agents devoted to
highly specific maneuvers such as cut-in or U-turn. It also
uses distributed Computing, in which ”the bubble mechanism
allows SMARTS to scale up without sacrificing interaction
realism” (Zhou et. al., 2020). Moreover, there is a need
to allocate computing resources according to the type of
simulation required, as social agents cannot be considered
as ”Non-character Players (NPCs)” as they require a huge
computing time. This gives rise to another issue of managing
the highly sophisticated deep learning dependencies of the
social agents, thus, here using the Ray approach, which is
a distributed system where AI applications can parallelly
learn from the interactions by continuously interacting with
the environment distributed among various computers. [3].
Therefore, through distributed computing, large-scale sim-
ulations wherein multiple computers can be dedicated for
performing specific tasks inside various bubbles [6], which,
makes proper utilization of resources and faster execution.
B. MADRaS : Multi Agent Driving Simulator
MADRaS is an open-source Multi-Agent DRiving Simula-
tor for motion planning in AD, which demonstrates its ability
to create driving scenarios with high degrees of variability
[5]. It mainly accomplishes demanding tasks such as ”driving
vehicles with drastically different dynamics, maneuvering
through a variety of track geometries at a high speed, navi-
gating through a narrow road avoiding collisions with both
moving and parked traffic cars and making two cars learn to
cooperate and pass through a traffic bottleneck” (Santara et.
al., 2020). It uses the concept of curriculum learning, where
the agents break the tasks into a hierarchy of subtasks and
fulfil each of the sub-tasks to achieve the main goal of the
particular scenario, Moreover, to minimize the computational
overhead for perception and action, it uses physics simulator
and representative graphics, in which each driving agent
gets a highlevel object-oriented representation of the world
as observation and an OpenAI gym (Brockman, Cheung,
Pettersson, Schneider, Schulman, Tang, and Zaremba, 2016)
interface for independent control [5]. From the architecture,
the TORCS simulator, which stands for The Open Racing
Car Simulator (Wymann et al., 2000) is capable of simulating
the necessary elements of vehicular dynamics such as mass,
rotational inertia, collision, mechanics of suspensions, links
and differentials, friction and aerodynamics as well as it
offers a variety of free tracks and cars for experimenting.
Furthermore, for enabling communication, it uses Simulated
Car Racing (SCR) server, which sets up a User Datagram
Protocol (UDP) communication (which is a wireless or
connection-less communication protocol) among all the cars
through their individual dedicated UDP port, from Fig. 2, the
double-headed arrows between various agents represents the
UDP communications between them. Moreover, MADRaS
consists of Traffic agents, which are agents formed using
a pre-programmed algorithm and inhibits pre-defined be-
haviours, which can be programmed through a common base
template by adding various interesting behavioural patterns
and are equipped with functions responsible for avoiding
collisions and going out of track [5]. However, these traffic
agents are different from that of the MADRaS agents, which
are based upon TORCS and are learning agents, which use
MARL to learn AD, based on the traffic agents and other
MADRaS agents’ behaviours. Overall, as seen from Fig.
2, the SCR client through UDP as discussed before, talks
to the TORCS server through a dedicated port, where the
simulation engine as a whole is a UDP server, and all the
other agents are UDP client, thus, establishing a connection
between all learning and traffic agents.
For the exposure of agents to different types of scenarios,
MADRaS offers various tracks and car models inherited
from the TORCS platforms. Moreover, the traffic agents are
of varied types showing behaviours like [5]:
 Fig. 2.
• Driving on a specified lane at a specified speed
• Speed varying in terms of Sine wave, i.e. decreasing
and increasing speeds at specific time intervals.
• Agents randomly changing the lanes
• Agent that drives specific distances and subsequently
parks itself
• Agents randomly stopping while driving
Furthermore, MADRaS uses a Snake oil library of Gym-
TORCS (Yoshida, 2016) [5], which is an OpenAI Gym
(Brockman et al., 2016) wrapper for SCR cars developed to
operate with Reinforcement Learning experiments. MADRaS
is built upon GymTORCS by increasing its stability, sim-
plicity in usage and extra features like multi-agent training
and custom traffic cars. The Snake oil library is responsible
for parsing state information returned by the TORCS server
about each agent, like odometry, range data, obstacle detec-
tion, engine statistics and metadata regarding the position of
a particular vehicle relative to the other cars on the road [5].
Also, it commands agents to control their respective cars via
steering, acceleration and brake commands.
C. MACAD-Gym: Multi Agent Connected Autonomous
Driving
The study performs three studies in order to answer the
study topic of how modifications in a robot’s appearance and
behaviour affect people’s reactions to them. The researchers
constructed a variety of face characteristics for robots in
order to test human reactions to both humorous and serious
robot behaviours for various tasks.The appearance and social
behaviour of a robot should either match the seriousness of
the task and situation, or be based on the positive hypothesis,
which states that the more attractive a robot appears and
the more extraverted and cheerful its behaviour is, the more
MADRaS stimulator
people will accept and comply with the robot, or the match-
ing hypothesis, which states that the appearance and social
behaviour of a robot should match the seriousness of the task
and situation.This paper finds that in the real world, robots
based on the matching hypothesis will prove to be much
more cooperative, especially in many day-to-day serious
work activities, such as taking medicines, exercising, and
following a healthy routine, where people will be required to
complete these tasks.Serious robots are more likely to follow
the orders of a serious robot in a disciplined manner than
entertaining robots.
For their first experiment, the researchers performed an
online study with 108 pupils. They made 12 2D robots that
were classified by levels of human similarity (more human-
like, midway, more robot-like), age (youth and old), and
gender (male, female) [4], and the students had to allocate
the robots to a variety of professions from an inventory.
The participants were asked to assign an appropriate robot
to each profession from a list of 12 options, based on
how well that robot would fit in the workplace or among
humans. perform better and save a greater number of human
lives. In the sphere of socially engaged jobs, on the other
hand, an engaging playful human-like robot will accomplish
wonders. The results were specified for many jobs that were
social, like actress and drawing instructor (Artistic), retail
clerk and sales representative (Enterprise), office clerk and
hospital message and food carrier (Conventional), aerobics
instructor and museum guide (Social) [4], human-like robots
were preferred over machine ones. However, for jobs of a
lab assistant, customs inspector, soldier and security guard,
machine-like robots were preferred. the rationale being that
the latter jobs don’t require many interactions with humans,
moreover as a robot in such jobs, are far better than humans,
as humans security guards tend to sleep at nighttime, customs
 Fig. 3. MACAD-Gym stimulator
inspector might take longer to search out a controversy than
a person’s and a military robot in situ of a soldier would
perform better and would save more human lives. On the
opposite hand, an interactive playful human-like robot will
do wonders within the fields of socially active jobs.
The second study measured human responses for 2 per-
sonalities of a robot which were, playful and heavy, where,
the playful robot had humorous, friendly conversations and
therefore the serious robot had more straightforward, onpoint
conversations with the participants. The experiments were
through with both the robots and these robots asked the
participants to try to to some exercise routines in pre-defined
dialogue conversations, where the playful robot told jokes,
laughed and made the conversation humorous and friendly,
whereas the intense robot had an easy on-point conversation
and surprisingly, the results were specified the participants
exercised longer with the intense robot. Moreover, they rated
the intense robot as significantly more than the playful robot
in terms of conscientiousness and smartness, but less playful
and witty than the playful robot. They also rated the playful
robot as slightly more obnoxious. This study clearly meant
that for accomplishing more important tasks interactions with
the intense robot would be more beneficial.
Finally, the third trial was an extension of the second,
in which participants were required to complete two tasks,
one of which was comparable to the prior exercise and
the other of which was related to jellybeans. Participants
in the jellybeans job were instructed to guess the flavours
and, as a result, create as many concoctions as possible
by combining different jellybeans. They were also asked
to conduct activities for as long as they could in order
to compare the two trials. Participants in the study spent
more time on the jellybean activity because it was more
pleasurable.Furthermore, under the jellybean condition, the
fun robot evoked more compliance than the serious robot, but
the serious robot elicited more compliance than the playful
robot.In the exercise situation, the amusing robot performed
well. The most compliance was generated by the fun robot
in the jellybean condition overall.
To recapitulate, the robots’ personalities and appearances
had an impact on the participants’ mental models as well
as their desire to follow the robots’ commands. However,
the participants were more predisposed to the idea that the
robots should behave and look in accordance with their
principal objective of job completion, therefore appearances
and behaviours were just secondary. Thus, the robot whose
personality corresponded to the participants’ expectations in
terms of task seriousness or playfulness yielded the greatest
outcomes in terms of participant compliance with the robot.
D. BARK: Behaviour BenchmARK
MACAD-Gym is a multi-agent reinforcement learning
platform with the support of Connected Autonomous Driving
(CAD) simulation. It consists of MACAD-Agent, having a
set of baseline/starter agents, which enables the community
to add on various algorithms upon the baseline agent’s
algorithm, to conduct various experiments as well as to
train multiple agents which can be contributed and thus,
can be experimented using the platform. Here, the baseline
algorithm is trained in a partially observable environment,
with the scenario being 3-way urban intersection and stop-
sign controlled. Moreover, the simulations use raw, cam-
era observations to learn the behaviours of various agents’
strategies and actions. In the MACAD-Gym simulator, the
goal of each agent is to maximize the expected value of its
long-term future reward, where, unlike single-agent systems,
the objective function of an agent depends on the policies
of the other agents in the multi-agent scenario (Praveen,
2016). Thus, the optimal policy of the agent is the best
response depending on the policies of other agents present
in the environment. The agents follow different policies,
which are shown in Fig. 3, they can be centralized learners
or decentralized learners. In centralized learning, the actors
of each vehicle have their own previous experience (XP)
memory, which stores the baseline algorithms as well as
any newly learnt strategies into its memory. All the agents
thus, share those memorized strategies, like parameters,
trajectory sequences, etc. among each other via Vehicle-to-
Vehicle (V2V), Vehicle-to-Infrastructure (V2I) or Vehicleto-
Pedestrians (V2P) depending upon the type of agents and the
environment. This shared information is fed into a centralized
learner algorithm, which then decides a global optimal
 Fig. 4.
solution for the environment as a whole and commands
individual agents to follow the specific strategy to achieve
the globally optimal solution. Contrary to the centralized
learners, the decentralized learners’ (Fig. 3 right) aim is to
achieve the locally optimal outcome. Here, each vehicle has
separate learning algorithms and memories, through which
each agent can learn locally and thereafter, can share the
learned strategies and actions to the common XP memory
where the results are stored and actions are performed.
Hence, here there is no common learning feature, thus
making learning decentralized, making the agents achieve
a local optimal output, rather than a global optimal solution.
• World and ObservedWorld Model: TheWorld model
of BARK consists of the map, static and dynamic
objects (agents). The ObservedWorld model, as the
name suggests reflects the world which is perceived
by the agents, where different degrees of observability
can be represented by either completely restricting
access to the world behaviour model or only allowing
consideration of ObservedWorld parameters which
consists of occlusions and sensor noise [1].
• Agent models: There are two interfaces in terms of
agent model, one responsible for calling a behaviour
model and thereafter, generating behavioural trajectory.
And another for executing an agent-specific model
for determining the next state, based on the generated
behavioural trajectory [1]. For accomplishing the above
BARK architecture
two objectives, some additional information is required,
such as (a) Goal Definition: Each agent has a specific
goal, like geometric goal regions or lanebased goals.
(b) Road corridor: Agents use it for determining the
set of roads, lanes, topology information about road
connection, etc. (c) Polygon: A 2D polygon which
specifies a particular shape of the agents
• Scenario: It contains a list of agent and their behaviours,
initial states, goal definitions, execution models, and
a stimulation map. Thus, in a particular scenario the
agents, using the factors contained in the Scenario, tend
to learn and stimulate both interactive and data-driven
scenarios
• Scenario generation: It contains source-sink pairs which
determine a lane where the agents are present. It also
has other parameter sets which specify the distribution
of agent states, their behaviours, goal definitions and
execution models. Hence, a scenario using the above
factors is generated and then executed, allowing to
model a variety of scenario types and easy extension
of already present scenarios
• Benchmarking: It provides benchmarking dataset,
which consists of large-scale behaviour models and
scenario sets and BenchmarkRunner, which evaluates
the specific behaviour models and parameters present
 in the benchmarking database [1]. The evaluations are
based on:
– StepCount: Counts the no. of steps
– GoalReached: Returns binary digit representing
goal satisfied or not
– DrivableArea: checks if the agent is inside the
RoadCorridor or not
– Collision: checks if any collisions have occurred
– GoalDistance: Calculates Euclidean distance as per
the GoalDefintion
The results of these experiments showed that human’s expec-
tations for robot’s movements will change based on different
environmental constraints, and the participant’s predictions
of robot’s choices gave different insights about what they
think about the robot’s capabilities. Moreover, the robot’s
appearance also played a huge role in determining its move-
ments and behaviours.
III. BACKGROUND
Multiple aspects, such as vehicles operated by a variety of
agents, settings, tasks, and scenarios, are used in the MARL-
based AD to achieve the goal of self-driving cars engaging
with other agents and working with one another to make the
overall scene of AD successful. This section examines the
numerous sorts of environments, tasks, and agents available
in the most recent simulators for experimentation.
A. Types of environment
1) Fully/Partially Observable environment: For the envi-
ronment to be fully observable, all agents are required to
observe the whole map of the environment, including all the
other agents like vehicles, pedestrians, as well as traffic lights
and cross roads behaving as agents, as a whole at each and
every point of time [4]. However, most of the environments
in the real-world scenario are inherently partially-observable,
as no agent at any particular time cannot observe the entire
environment, as wireless communications are limited to
certain areas, thus, making connectivity between vehicles or
any other agents, both helpful as well as limited [4].
2) Synchronous/Asynchronous: Synchronous as the name
suggests, is an environment where all the agents act simulta-
neously at a particular moment, whereas in an asynchronous
environment all the agents act at different time [4]. For
instance, the vehicle agents either stopping or moving, while
the pedestrians acting similarly at the same time based on
the same traffic light indication can be considered as actions
done in a synchronous manner, whereas while making turns,
the agent on a single lane behind an another agent need to
wait till the agent ahead completes it’s actions. Here, the
environment is asynchronous.
3) Adversarial: Adversarial in terms of environment are
different factors like weather, and altered communication
medium (which can caused by any unanticipated attacks)
[4], which causes hindrance in the normal functioning of
the agents. This type of environmental nature can be dealt
by using extra conditions in the algorithm which can subse-
quently overcome any attacks by taking immediate measures
as well as can figure out bad weather conditions and con-
sequently take other routes or change strategies which can
be most favourable at the moment based on the respective
environmental circumstances.
B. Types of agents
1) Homogeneous agents: Those are the agents belonging
to same category i.e. agents handling only cars or only
motorcyclists [4]. Thus, as the types of agents are same, their
action spaces are likely to be similar due to same physical
properties. Also, as homogeneous agents are collection of
only same agent types, their interactions and communications
are limited of being made between same homogeneous
agents [4], which leads to not considering other agents like
traffic lights, pedestrians or motor cycles, by a homogeneous
car agent, making the simulator limited.
2) Heterogeneous agents: Contrary to homogeneous
agents, heterogeneous agents considers various different
types of agents interacting with each other [4]. Mostly, all the
realistic applications are composed of heterogeneous agents,
for example, multiple vehicles, pedestrians, and even traffic
lights (can be represented as an intelligent actor) capable of
interacting with each other are heterogeneous agents.‘
3) Communicating agents: As the name suggests, agents
that are capable of communicating with each other with
the sole purpose of increasing information availability [4],
by means of either direct or indirect channels, which can
be either one way communication between traffic light
and vehicles or two ways between two or more vehicles
when negotiating in the traffic. The communication can be
established via virtual/shared/crowd sourced sensors, hand
and body gestures by the pedestrians and visual external
displays like car light signals, especially used while turning
and braking, traffic light indication signals, auditory signals
like horns, ambulance and police car sirens, and many more.
4) Non-communicating agents: Here if the type of agent
is such that it can not communicate either because of the
incapability of communications amongst agents or due to the
nature of agent, forcing it of not using the communication
channels.
C. Nature of tasks
1) Independent tasks: This type of tasks consists of Self-
Interested agents, who are not selfish (only care about things
that are beneficial for them) or malicious (harm other agents),
but are likely to follow the description of states, which
it prefers, thus, their actions are highly motivated by this
description (i.e. the algorithm). The model of these types
of agents are similar to single-agent environment, mainly
having their own unique goals, and having no explicit com-
munication mediums [4]. Thus, all communication deprived
individual single agents will accomplish their own objectives,
and ”will benefit from agents modeling agents” (Praveen,
2019).
 2) Cooperative tasks: In this type of task all the agents act
as a cooperative unit, which requires developing algorithms
for different agents in a way by which they can learn to
achieve a near-globally optimal outcome for each of the
agents present in a particular scenario. Thus, this task leads
to develop agents which communicate with each other and
therefore, can solve and subsequently eliminate issues like
congestion and collisions by having pre-planned communi-
cation for solving such future difficulties that might occur
[4].
3) Competitive tasks: As opposed to that of Cooperative
tasks, agents here are developed based on a competitive
nature specifically for extreme driving scenarios like road-
rages. This type of task leads to the formation of Zero-sum
stochastic games, where 2 agents have opposite interests and
those agents can be used for law enforcement or development
of strong adversarial (External environmental hinderances)
agents, capable of handling actions in a competitive task.
4) Mixed: As per the name, in mixed tasks agents tend
to behave in a cooperative, independent and competitive
manner as per the particular task the agent needs to act.
This type is the most beneficial among all others as it leads
to maximizing rewards. Moreover, an agent designed to be
of a particular nature (say independent), can be extrapolated
to learn other tasks (cooperative or competitive), making the
simulation highly efficient as the best possible task can be
easily executed by that particular agent having mixed tasks.
IV. RESULTS
All of the above-mentioned comparison criteria, as well
as details about the type of environment, tasks, and agents,
as described in background knowledge, are collectively used
for all simulators explained in the Related work section, and
a brief overall comparison and contrast about each specific
term is presented in Table.
V. DISCUSSION
When distinguishing the four simulations discussed
above, there are several criteria that are common to all of
them that can be used to make useful comparisons, such
as determining which simulation gives good results for one
specific criterion and then using that simulator to experiment
with related features.
1) Agents: For comparison, various agent types
outlined in the Background section (section 3) are used.
Agents can be homogeneous or heterogeneous, learning
or non-learning, communicative or non-communicating,
synchronous/asynchronous or mixed, independent or
cooperative or competitive, and so on. In the Results
section, all of the above attributes of a given agent that
are employed in each simulator are listed and compared to
other simulators.
2) Scenarios: Because a simulation necessitates
particular circumstances in which the agents can act as
they should, this section compares the scenarios included
in each simulator. Two-way or three-way traffic, as well as
intersections, are the most typical types of scenarios, and
the actions that agents contemplate when dealing with them
can be used to evaluate the simulator’s overall effectiveness.
3) Involvement of humans: This is the most significant
component since, in a real-world scenario, there is a good
chance that there will be more people than multi-agent-based
AD, which may or may not behave as expected by AD cars
(maybe humans are unwilling to communicate with AD cars
before or after executing activities).As a result, whether or
not the simulators use human-like agents to train AD cars
is a significant consideration
4) Communication techniques: When it comes to
dealing with AD automobiles, communication between
multiple agents is critical. There are several methods for
facilitating communication among agents, but they all rely
on wireless communication because cable connection is
difficult to implement in a real-world environment. As a
result, those simulators use diverse server and client models
based on IP addresses, WIFI, and Bluetooth technologies
for wireless communication.
5) Environment: This primarily concerns the type of
environment, which is divided into two categories. There
are two types of observable: partially observable and fully
observable. In comparison to fully view-able environments,
partially observable environments are more frequent because
establishing close connections for communication is a
simpler and less expensive choice, where each agent is
required to have a broad understanding of the environment,
which is a difficult challenge to execute in a real-world
setting.
6) Actions: The agents existing in the environment must
follow specific pre-defined activities in order to achieve
their objectives dependent on the circumstances they are
in. As a result, all agents have a a list of appropriate
activities, primarily turning, braking, and speeding, which
they might use to behave based on the situations that likely
to to appear when driving in order to prevent colliding with
other vehicles Objects that can be found in the environment.
7) Rewards: Because AD is founded on RL, it employs
the concept of rewards. In real life, to get agents to learn
certain things, a method of rewarding them (in the form of
points) if they perform the actions correctly and penalising
them (in the form of penalties) if they clash or detour from
their core goals connected to completing a task is used. As a
result, this section discusses the simulators’ reward functions
as well as some of the criteria they employ to determine the
appropriate payouts.
 VI. CONCLUSION Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexander
Cowen Rivers, Zheng Tian, Daniel Palenicek, Haitham bou Ammar,
Certain aspects of all four simulators are very similar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. 2020.
while others are diametrically opposed. To begin with, SMARTS: Scalable Multi-Agent Reinforcement Learning Training
SMARTS, MADRaS, and BARK support a variety of agents, School for Autonomous Driving. arXiv:arXiv:2010.09776
[7] AdaFruit, 2016. Arduino Complex. International Conference on
some of which are pre-defined and non-learning, and others Robotics and Automation. 2767–2772. https://cdn-shop.
which are powered by algorithms and make decisions based adafruit.com
on RL. MACADGym, on the other hand, only employs pre- [8] Anon., 2018. Dude, Where’s My Autonomous Car? The 6
Levels of Vehicle Autonomy https://www.synopsys.com/
defined learning agents and has no more than two variations automotive/autonomous-driving-levels.html
of agents. Non-learning agents are extremely useful in a [9] Anon., n.d. The path to autonomousdriving.
simulator because they can easily be mistaken for human [10] BENUI, N., 2012. Weigh-in-motion system. http://www.
loadcell.cn/weigh-in-motion-system.html
agents who are unwilling to learn, making the simulator [11] Davies, A., 2018. The WIRED Guide to Self-Driving Cars. 576–577.
better from a real-world standpoint.Furthermore, all of the https://www.wired.com/story/guide-self-driving-cars/
simulators have very identical situations and environment
characteristics. All other simulators, with the exception of
MACAD-Gym, only examine a partially observable environ-
ment, whereas MACAD-Gym also considers a fully viewable
world, which is harder to set up.Furthermore, all of the
simulators’ rewards and behaviours are very identical, with
the exception of one.
Braking, accelerating, and turning are examples of actions,
whereas rewards are examples of rewards.
Penalties and other progress incentives are combined into
a weighted total. As a result,
Finally, for more scenarios and agents to choose from,
SMARTS
can be the best solution, however for experimenting in a
small space,
environment that is completely viewable, with additional
action options and
MACAD-Gym can communicate using simple UDP-based
communication protocols.
be taken into account Furthermore, MADRaS would de-
liver the greatest results for developing racing scenarios in
which various racing tracks are provided for experimentation.
Finally, BARK, which emphasises the importance of social
agents in a similar way to SMARTS, is a suitable alternative
for newcomers in this field because it allows them to play
more with non-learning social agents by writing pre-defined
algorithms based on AD.
R EFERENCES
[1] Julian Bernhard, Klemens Esterle, Patrick Hart, and Tobias Kessler.
2020. BARK: Open Behavior Benchmarking in Multi-Agent Environ-
ments. https://doi.org/10. 1109/IROS45743.2020.9341222
[2] D. Krajzewicz, G. Hertkorn, C. Rossel, and P. Wagner. 2002. SUMO
(Simulation of Urban MObility) an open-source traffic simulation. In
MESM. 183–187.
[3] P. Moritz, R. Nishihara, S.Wang, A. Tumanov, R. Liaw, E. Liang, M.
Elibol, Z. Yang, W. Paul, M. I. Jordan, et al. 2018. Ray: A distributed
framework for emerging AI applications. In OSDI. 561–577. https:
//doi.org/10.1177/1541931215591337
[4] Praveen Palanisamy. 2019. Multi-Agent Connected Autonomous Driv-
ing using Deep Reinforcement Learning. arXiv:arXiv:1911.04175
[5] Anirban Santara, Sohan Rudra, Sree Aditya Buridi, Meha Kaushik,
Abhishek Naik, Bharat Kaul, and Balaraman Ravindran. 2020.
MADRaS : Multi Agent Driving Simulator. arXiv:arXiv:2010.00993
[6] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu,
Jiayu Miao,Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng
Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel
Graves, Dong Chen, Zhengbang Zhu, Nhat Nguyen, Mohamed El-
sayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu,
Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani,