See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/357643577

Reinforcement Learning for Scheduling and Mimo beam Selection using

Caviar Simulations

Conference Paper · December 2021

DOI: 10.23919/ITUK53220.2021.9662100

CITATION READS

1 13

9 authors, including:

Ailton Oliveira Felipe Bastos

Federal University of Pará Federal University of Pará
14 PUBLICATIONS   16 CITATIONS    8 PUBLICATIONS   5 CITATIONS   

SEE PROFILE SEE PROFILE

Daniel Takashi Ne Do Nascimento Suzuki Cleverson Nahum

Federal University of Pará Federal University of Pará
6 PUBLICATIONS   1 CITATION    19 PUBLICATIONS   22 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

SAMURAI - Smart 5G Core And MUltiRAn Integration View project

All content following this page was uploaded by Ailton Oliveira on 10 February 2022.

The user has requested enhancement of the downloaded file.

 REINFORCEMENT LEARNING FOR SCHEDULING AND MIMO BEAM SELECTION
USING CAVIAR SIMULATIONS

João Paulo Tavares Borges1 ; Ailton Pinto de Oliveira1 ; Felipe Henrique Bastos e Bastos1 ; Daniel Takashi Né do Nascimento
Suzuki1 ; Emerson Santos de Oliveira Junior1 ; Lucas Matni Bezerra2 ; Cleverson Veloso Nahum1 ; Pedro dos Santos Batista3 ;
Aldebaro Barreto da Rocha Klautau Júnior1
1 Universidade Federal do Pará, Belém 66075-110, Brazil
2 Universidade Estácio de Sá, Belém 66055-260, Brazil
3 Ericsson Research, 164 80 Stockholm, Sweden

ABSTRACT

This
paper
describes
a
framework
for
research
on

Reinforcement
Learning
(RL)
applied
to
scheduling
and

MIMO
beam
selection.
This
framework
consists
of
asking

2021 ITU Kaleidoscope: Connecting Physical and Virtual Worlds (ITU K) | 978-92-61-33881-7/21/$31.00 ©202110.23919/ITUK53220.2021.9662100

the
RL
agent
to
schedule
a
user
and
then
choose
the
index
UAV

of
a
beamforming
codebook
to
serve
it.
A
key
aspect
of
this

problem
is
that
the
simulation
of
the
communication
system

and
the
artificial
intelligence
engine
is
based
on
a
virtual

world
created
with
AirSim
and
the
Unreal
Engine.
These

components
enable
the
so-called
CAVIAR
methodology,

which
leads
to
highly
realistic
3D
scenarios.
This
paper

describes
the
communication
and
RL
modeling
adopted
in

the
framework
and
also
presents
statistics
concerning
the

implemented
RL
environment,
such
as
data
traffic,
as
well
as

results
for
three
baseline
systems. Figure
1
–
CAVIAR
simulation
scenario,
depicting
the

radiation
pattern
(in
light
green)
corresponding
to
the
chosen

Keywords
-
5G,
6G,
beam
selection,
MIMO,
mmWave,
RL beamforming
codebook
index
to
serve
a
drone
(at
the
right).

1. INTRODUCTION
Reinforcement
Learning
(RL)
is
a
learning
paradigm
suitable

for
problems
in
which
an
agent
has
to
maximize
a
given

reward,
while
interacting
with
an
ever-changing
environment.

This
class
of
problem
appears
in
several
points
of
interest

in
5th
Generation
(5G)
and
6th
Generation
(6G)
mobile

networks,
such
as:
congestion
control
[1],
network
slicing
[2],

resource
allocation
[3],
and
the
5G
Physical
Layer
(PHY)
[4].

However,
the
lack
of
freely
available
data
sets
or
environments

to
train
and
assess
RL
agents
is
a
practical
obstacle
that
delays

the
widespread
adoption
of
RL
in
5G
and
future
networks.
To
address
this
challenge,
some
works
explore
the
use
of

virtual
worlds
to
generate
data
sets
by
creating
environments

for
communications
in
general
[5],
and
Artificial
Intelligence

(AI)
/
Machine
Learning
(ML)
applied
to
5G/6G
[6],

leveraging
the
fact
that
5G
and
beyond
systems
will
benefit

from
rich
contextual
information
to
improve
performance

and
reduce
loss
of
radio
resources
to
support
its
services

[4,
7,
8].
So,
the
key
idea
in
this
paper
is
to
use
realistic

representations
of
deployment
sites
together
with
physics

and
sensor
simulations,
to
generate
a
virtual
representation

that
combined
with
the
communication
network
simulator,

enables
training
RL
agents
for
tasks
such
as
beam
selection.
Systems
such
as
IEEE
802.11ad
are
usually
designed
for

worst-case
scenarios
and,
in
most
situations,
continuously

send
signals
that
do
not
carry
information
(overhead)
[9].

This
overhead
may
represent
a
significant
p arcel
o f
the

channel
capacity,
and
decreasing
it
is
a
fundamental
problem

that
can
enable
systems
to
improve
the
usage
of
physical

resources
(e.g.,
with
lower
latency
and
higher
bit
rates)
[10,
11,
12].
In
this
work,
the
beam
selection
and
user
scheduling

problems
are
posed
as
a
game
that
must
be
solved
with

RL.
The
game
is
based
on
a
simulation
methodology

named
Communication
Networks,
Artificial
Intelligence
and

Computer
Vision
with
3D
Computer-Generated
Imagery

(CAVIAR),
with
a
preliminary
version
proposed
in
[13].

The
CAVIAR
simulation
integrates
three
subsystems:
the

communication
system,
the
AI
and
ML
models,
and
finally

the
virtual
world
components.
In
this
paper,
the
problem
is

based
on
simulating
a
communication
system
immersed
in

a
virtual
world
created
with
AirSim
[14]
and
Unreal
Engine
[15].
More
specifically,
the
goal
is
to
schedule
and
allocate

resources
to
Unmanned
Aerial
Vehicles
(UAVs),
cars
and

pedestrians,
composing
a
scenario
with
aerial
and
terrestrial

978-92-61-33881-7/CFP2168P @ ITU 2021 Kaleidoscope

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.
 2021 ITU Kaleidoscope Academic Conference

Figure 2 – CAVIAR simulation overview.

User Equipment (UE). The RL agent is executed at the files, which are named episodes, containing the trajectory
Base Station (BS) and periodically takes actions based on the data of all moving objects within a simulation. To generate it,
information captured from the environment, which includes a waypoint file, which is a text file with reference points, must
channel estimates, buffer status, and positions from a Global be executed by AirSim. During its execution, the information
Navigation Satellite System, such as GPS. The RL agent from the mobile elements is stored in the episode. Each
receives a reward based on the service provided to the episode lasts about three minutes, with a sampling interval
users. The training occurs “offline”, without rendering the of ten milliseconds, and is composed by columns related to
3D scenes, but it is possible to render the output in a position and orientation for pedestrians and cars, with the
post-processing step and generate a video. addition of acceleration, linear, and angular velocities for
UAVs. To use the episode files to obtain information from
This work is organized as follows. In Section 2 we discuss Multiple-Input Multiple-Output (MIMO) channels and data
CAVIAR simulations in general and the specific RL problem traffic, one must execute them within the CAVIAR simulation
addressed in this paper. Sections 3 and 4 describe the environment.
communication and machine learning models, respectively.
Simulations results are presented in Section 5, while Section Unreal/AirSim
6 concludes the paper.
Waypoint
Episodes
generator
2. CAVIAR SIMULATIONS
- MIMO channels
As proposed in [6] and shown in Figure 2, the CAVIAR SimulaƟon
- Combined channel magnitudes
framework incorporates three subsystems: AI/ML, virtual - Data traĸc environment
world, and wireless communications. In the following
paragraphs we describe the framework, focusing first on the Figure 3 – CAVIAR data generation.
overall description of the methodology and then on how
it was realized in the user scheduling and beam selection
environment. 2.1 Overall CAVIAR description

RL tasks can be continuous or episodic; this last category As previously mentioned, Figure 2 displays an overview of the
assumes the context adopted in this work. Figure 3 expected components in a CAVIAR simulation. In summary,
exemplifies the CAVIAR data generation pipeline: the data the blocks encompassing the proposed simulation strategy
set is provided as a set of Comma-Separated Values (CSVs) can be described as follows: the Communications Engine

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.

handles all information regarding the communication aspect
of the simulation, such as data traffic, buffer and channel
information. In the CAVIAR simulations, channels can be
pre-computed and the communication simulation decoupled
from the physics engine, as often used in AI/ML applied to
beam selection [7, 4].
The 3D assets used in the Environment and as Mobile entities,
such as UAVs, cars, buildings, etc, are either created or
obtained online, as described in [4]. They compose the
simulation environment as fixed or mobile objects, whose
eventual movements and interactions are managed by the
Mobility engine and by the Physics engine of the virtual
world subsystem, respectively. The Sensors engine output
constitutes the input to the AI/ML frontend engine.
The AI/ML engine receives signals and communication
parameters that were simulated in the virtual-world
environment and suggests actions that are then implemented
by the Orchestrator, that also considers parameters from the
Communications engine and Environment. An example of an
action in the context of beam selection would be sending to
the base station a list of codebook indices to try and avoid a
full beam sweeping.
2.2 CAVIAR simulation for user scheduling and beam
selection problems
For the user scheduling and beam selection problems, the
Communications engine used by CAVIAR simulations was
defined by geometric channel models, further described in
Section 3. The Physics engine and the Mobility engine are
handled by Unreal Engine and AirSim, and finally, for the
AI/ML engine, we assume an RL environment. Upon having
the episodes available, the environment can be executed
to allow an agent to assume the role of a BS, scheduling
and serving the users with a specific beam chosen from its
codebook. The episodes can be used to train and test agents,
as well as store the outputs in a CSV file, which can be used
as it is or reproduced graphically in Unreal, as described in
Figure 4.
Unreal/AirSim
Waypoint
Episodes
generator
SimulaƟon
environment
Rendering
Agent output file
RL agent RL agent
(scheduled user and
codebook index) test training
Figure 4 – CAVIAR simulation flow.
Using a virtual scenario provided by CAVIAR, three mobile
entities: a pedestrian, a car, and a UAV are simulated in
order to generate a data set of urban mobility. This data
is organized in episodes that contains spatial information
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.
Connecting physical and virtual worlds
(position, orientation, acceleration, etc.) of each mobile
entity. For this problem, the samples are collected at
every 10ms and they contain information of 37 entities (34
pedestrians, 2 cars, and 1 UAV).
As shown in Figure 4 the spatial data generated by the
virtual scenario is the input for the CAVIAR simulation
environment, more specifically, the communication engine,
which is responsible for computing the radio channels and
other parameters related to the telecommunication system,
such as buffer size, etc. The output of the communication
engine along with the spatial data is the input for an RL
agent, that is trained to choose, in each time slot, a user to
serve, and the beam that should be used.
3. COMMUNICATION MODEL
The simulation environment incorporates not only the
RL-related functions required by the OpenAI Gym
Application Programming Interface (API), but also the
functions related to the communication system. This section
will describe the adopted communication model.
We assume downlink transmission using a carrier frequency
fc = 60 GHz and a bandwidth of 100 MHz. The BS serves
three distinct receivers or UEs, which are located at a car, at
a UAV and used by a pedestrian. The BS has an individual
buffer with a size of 1 Gb for each receiver. We assume that
each packet has 8188 bytes. When the buffer of a specific
user becomes full, the newly arrived packets are dropped.
The MIMO system corresponds to an analog architecture
using a Uniform Planar Array (UPA) with Nt antenna
elements at the BS, and receivers with UPAs with Nr
antennas. Therefore, the MIMO channel between the BS
and a given user is represented by a Nr × Nt matrix H. The
codebooks are obtained from Discrete Fourier Transform
(DFT) matrices and denoted by Ct = {w̄1, · · · , w̄ Nt } and
Cr = {f¯1, · · · , f̄ Nr }. They are used at the transmitter and
the receiver sides, respectively. When modeling the beam
selection, the chosen beam pair [p, q] is represented by a
unique index i ∈ {1, 2, · · · , M }, where M ≤ Nt Nr . Each
index p (or q) generates a specific radiation pattern, as
depicted in Figure 5 for a Nt = 64.
For the i-th index, the equivalent channel is calculated as:
yi = wi∗ Hfi, (1)
and the optimal beam index ˆi is given by
ˆi = arg max |yi |. (2)
i ∈ {1, ··· , M }
We do not take noise into account in order to isolate the
impact of the beam selection procedure.
Ray Tracing (RT) was used in [4], to generate realistic
communication channels H. For this paper, we have not used
RT but a simpler procedure based on the geometric MIMO
 2021 ITU Kaleidoscope Academic Conference

different network load scenarios representing a light and

heavy network traffic to simulate the traffic variations along
with the scenes. The simulation alternates light and heavy
traffic in each 1000 scenes. The total throughput of the
heavy scenario is bigger than the lighter one. Each user
has a specific traffic magnitude defined as a percentage of
total throughput in accordance with Table 1, enabling the
differentiation of applications. Figure 6 shows the histogram
of traffic throughput for each user in Gbps. Each user presents
the heavy and light traffic behavior. The incoming traffic for
each user is buffered when there is buffer space available,
otherwise the excess of packets are tail-dropped. The packets
are also dropped when they occupy the buffer for more than
10 seconds.
Table 1 – Network load information for light and heavy
scenarios.
Figure 5 – Example of radiation pattern for specific beam
index with an 8 × 8 UPA. Network load Total throughput UAV (%) Pedestrian (%) Car (%)
Light 0.48 Gbps 50% 20% 30%
channel model [16]. The reason for this choice is that we first Heavy 0.96 Gbps 50% 20% 30%
want to evolve the AI/ML engine such that newer versions
allow rendering in (near) real-time, along the training of the
RL agent. Right now, we are not able to render each scene
and the agent choices, but CAVIAR-v2 is being developed
with this goal. We will later work in the CAVIAR-RT-v1,
where RT stands for support to ray tracing.

The simplified H currently represents a Line-of-Sight (LoS)

channel. A narrowband channel model [16] is used, but
wideband models can be readily incorporated in case their
extra computational cost is not an issue. For simplicity, the
users have a single antenna (Nr = 1) while the BS has a 8 × 8
UPA (Nt = 64).

The geometric channel model [16] is adopted with L = 2 Figure 6 – Histogram of packets traffic received by the BS
Multipath Components (MPCs): for each user.
 
L
H= Nt Nr α ar (φA, θ A)a∗t (φD, θ D ). (3)
4. MACHINE LEARNING MODEL
=1

The parameters in Eq. (3) are obtained as follows. The 4.1 Evaluation of RL agents
phase of the complex-gain α is obtained from a uniform
To evaluate the RL agent, the return G over the test episodes
distribution with support [0, 2π]. For generating the
is used. The return G e for episode e is
magnitude |α |, first the distance d between the BS and
the given receiver is used to calculate the received power e

Ns
via the Friis equation [17]. The path loss is obtained from Ge = re [t], (4)
this equation and determines |α |, which decreases with t=1
d. The elevation φ and azimuth θ  angles, for departure
where Nse is the number of scenes in episode e. The
(e.g. φD ) and arrival (e.g. φA) are obtained from the
corresponding reward re [t] at discrete-time t is a weighted
orientation provided by the LoS path. The nominal LoS
sum of transmitted and discarded packets given by
angles are slightly changed by adding to them Gaussian
random variables with zero-mean and variance of 1 degree. Ptx [t] − 2Pd [t]
re [t] = , (5)
These angles are used to compose the steering vectors at and Pb [t]
ar .
where Ptx [t], Pd [t], and Pb [t] correspond, respectively, to the
total amount (summation for all users) of transmitted, dropped
3.1 Traffic model and buffered packets at time t. The reward re [t] is restricted
to the range −2 ≤ re [t] ≤ 1. At each time t, a single user can
The users’ data traffic is defined as Poisson processes with be served, but Pb [t] accounts for the number of packets in all
time-varying mean λu [t] for user u. We specified two three buffers. Hence, re [t] = 1 only if all buffered packages

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.

of the scheduled user are transmitted, while the buffers of the
other two users were empty.
4.2 Possible inputs to RL agents
The inputs (also known as states or observations) can
be selected both from information provided in CSV files
(position (x,y,z), velocities, etc.) or obtained from the
environment, such as the buffer state and channel information
for specific beam index previously chosen.
More specifically, the RL agent can use: the UEs geographic
position in X, Y, Z, with the origin of the coordinate system
being on the BS RL agent. Also, the UEs orientation in the
three rotation coordinates: the front and side roll angles, as
well as its rotation over its own axis. Besides that, there are
also dropped, transmitted and buffered packets. Finally, the
last two other available input features for the agent are the bit
rate and the channel magnitude at each step of the simulation.
Note that we assume the BS (more specifically, the RL agent)
does not know the best index ˆi. In practice, this would require
a full beam sweep, which is assumed to be unfeasible in our
model due to stringent time requirements. Similarly, given
that the RL agent chose user u and beam index j at time t,
it learns only the magnitude |y j | and the spectral efficiency
Su,t, j for this specific user and beam index at time t.
The channel throughput Tu,t, j = Su,t, j BW is obtained by
multiplying the spectral efficiency by the bandwidth BW and
indicates the maximum number of bits that can be transmitted.
An empirical factor is used to adjust Tu,t, j in order to define
the network load, such that, for the given input traffic, some
packets have to be dropped.
Algorithm 1 summarizes the steps for executing an already
trained RL agent.
Algorithm 1: High-level algorithm of the RL-based
scheduling and beam selection problem.
Initialization for a given episode e;
while t ≤ Nse do
1) Based on the number of bits in the buffers of
the users and other input information, RL agent
schedules user u and selects beam index i;
2) Environment calculates combined channel
magnitude |yi | and corresponding throughput Ti ;
3) The number of transmitted bits is
Ri = min(Tu,t, j ; bu ) ;
4) Update buffers;
5) Receive new packets;
6) Eventually drops packets;
7) Environment calculates rewards and updates its
state;
8) Update buffers again;
end
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.
Connecting physical and virtual worlds
4.3 Experiment description
We developed an experiment using CAVIAR for the problem
of scheduling and beam selection. Given that a complete
episode file contains information about all moving objects in
a scene (all pedestrians, cars, etc.), we simplified the data
generated by the simulation assuming that the beam selection
RL agent, named B-RL, only uses data from the three served
users (uav1, simulation_car1 and simulation_pedestrian1).
Figure 7 – Channel maximum throughput when using always
the best beam index ˆi and a simple scheduling strategy that
chooses users sequentially (1-2-3-1-2-3...), in a round-robin
fashion.
The following results are extracted from an Advantage Actor
Critic (A2C) agent from the Stable-Baselines [18], trained
with default parameters. The states of the agent are defined by
seven features: X, Y, Z, packets dropped, packets transmitted,
buffered packets and bit rate. The action space is composed
by a vector with two integers: a numeric identity of the
user being allocated at the specific timestamp, that can range
between [0, 2]; and the codebook index of the beam to be
used to serve it, which is an integer from the range [0, 63].
Finally, the reward used Eq. (5).
Because the RL agent was designed to play the role of a simple
example and not optimize performance, two other agents were
developed: B-Dummy and B-BeamOracle. The B-Dummy
agent assumes random action choices for both the scheduled
user and which beam index to use. The B-BeamOracle agent
follows a sequential user scheduling pattern (1-2-3-1-2-3,...)
in a round-robin fashion, and always uses the optimum beam
index ˆi for the selected user. In Figure 7 we characterize the
channel maximum throughput of this experiment when using
B-BeamOracle.
5. EXPERIMENT RESULTS
The CAVIAR environment was used to generate 70 episodes,
from which 50 were used for training the RL agent, and 20 for
testing. We present results for the three agents: B-Dummy,
B-BeamOracle and the RL-based A2C agent.
In Figure 8, it is possible to verify the switching at every 1000
 2021 ITU Kaleidoscope Academic Conference


Figure 8 – Reward obtained by the B-BeamOracle agent for
a given episode. The traffic load switches every 1000 time
steps between “heavy” and “light”.
samples, between the “heavy” and the “light” data traffic.
The sequential scheduling proves to be sufficient to attend
the demand in light traffic situations, however, for intense
traffic moments, even using the best beam index ˆi, without
proper scheduling, the performance of the reward tends to be
negative.
Figure 9 – Histogram of the total sum of rewards achieved in
the test episodes.
Figure 9 shows a reward histogram for different agents along
20 test episodes. As expected, the B-BeamOracle presents
the best performance, while the B-RL achieves performance
close to the B-Dummy, which simply uses random actions.
One reason for the bad performance of B-RL is the choice
of its input parameters. None of the seven features help
the agent to directly learn the user and beam index used
in its previous decision. Better modeling of the agent can
substantially improve its performance.
6. CONCLUSION
This paper presented a framework for research on RL
applied to scheduling and MIMO beam selection. Using
the framework, we provided statistics of an experiment in
which an RL agent faces the problems of user scheduling and
beam selection. The experiment allowed us to validate the
designed environment for RL training and testing. Future
development will focus on rendering the 3D scenarios while
training the RL agent, as well as using more realistic channels
via ray tracing.
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.
ACKNOWLEDGEMENTS
This work was supported in part by the Innovation Center,
Ericsson Telecomunicacões S.A., Brazil, CNPq and the
Capes Foundation.
REFERENCES
[1] I. Nascimento, R. Souza, S. Lins, A. Silva, and
A. Klautau, “Deep Reinforcement Learning Applied
to Congestion Control in Fronthaul Networks,”
Proceedings - 2019 IEEE Latin-American Conference
on Communications, LATINCOM 2019, pp. 1–6, 2019.
[2] Y. Kim and H. Lim, “Multi-agent reinforcement
learning-based resource management for end-to-end
network slicing,” IEEE Access, vol. 9, pp.
56 178–56 190, 2021.
[3] X. Wang and T. Zhang, “Reinforcement Learning
Based Resource Allocation for Network Slicing in 5G
C-RAN,” in 2019 Computing, Communications and IoT
Applications (ComComAp), 2019, pp. 106–111.
[4] A. Klautau, P. Batista, N. González-Prelcic, Y. Wang,
and R. W. Heath, “5G MIMO data for machine learning:
Application to beam-selection using deep learning,” in
2018 Information Theory and Applications Workshop
(ITA). IEEE, 2018, pp. 1–9.
[5] E. Egea-Lopez, F. Losilla, J. Pascual-Garcia, and J. M.
Molina-Garcia-Pardo, “Vehicular networks simulation
with realistic physics,” IEEE Access, vol. 7, pp.
44 021–44 036, 2019.
[6] A. Klautau, A. de Oliveira, I. P. Trindade, and
W. Alves, “Generating MIMO Channels For 6G Virtual
Worlds Using Ray-tracing Simulations,” arXiv preprint
arXiv:2106.05377, 2021.
[7] A. Klautau, N. González-Prelcic, and R. W. Heath,
“LIDAR data for deep learning-based mmWave
beam-selection,” IEEE Wireless Communications
Letters, vol. 8, no. 3, pp. 909–912, 2019.
[8] W. Jiang, B. Han, M. A. Habibi, and H. D. Schotten,
“The road towards 6G: A comprehensive survey,” IEEE
Open Journal of the Communications Society, vol. 2,
pp. 334–366, 2021.
[9] M. Giordani, M. Mezzavilla, C. N. Barati, S. Rangan,
and M. Zorzi, “Comparative analysis of initial access
techniques in 5G mmWave cellular networks,” in 2016
Annual Conference on Information Science and Systems
(CISS). IEEE, 2016, pp. 268–273.
[10] J. Choi, V. Va, N. Gonzalez-Prelcic, R. Daniels, C. R.
Bhat, and R. W. Heath, “Millimeter-wave vehicular
communication to support massive automotive
sensing,” IEEE Communications Magazine, vol. 54,
no. 12, pp. 160–167, 2016.
 Connecting physical and virtual worlds

<> J.
Kim
and
A.
F.
Molisch,
“Fast
millimeter-

wave
beam
training
with
receive
beamforming,”
Journal
of
Communications
and
Networks,
vol.
16,
no.
5,
pp.
512–522,
2014.
<> P.
Zhou,
X.
Fang,
Y.
Fang,
Y.
Long,
R.
He,
and
9 Han,
“Enhanced
random
access
and
beam
training
for
millimeter
wave
wireless
local
networks
with
high
user
density,”
IEEE
Transactions
on
Wireless
Communications,
vol.
16,
no.
12,
pp.
7760–7773,
2017.

<> A.
Oliveira,
F.
Bastos,
I.
Trindade,
W.
Frazão,

" Nascimento,
D.
Gomes,
F.
Müller,
and
A.
Klautau,
“Simulation
of
Machine
Learning-Based
6G
Systems
in
Virtual
Worlds,”
Submitted
to
ITU
Journal
on
Future
and
Evolving
Technologies,
2021.
<> “Welcome
to
AirSim,”
https://microsoft.github.io/
AirSim,
accessed:
2021-10-19.

<> “The
most
powerful
real-time
3D
creation
tool,”
https:
//www.unrealengine.com,
accessed:
2021-10-19.
<> R.
W.
Heath,
N.
González-Prelcic,
S.
Rangan,
W.
Roh,
and
A.
M.
Sayeed,
“An
Overview
of
Signal
Processing
Techniques
for
Millimeter
Wave
MIMO
Systems,”
vol.
10,
no.
3,
pp.
436–453,
Apr.
2016.

<> Balanis,
Constantine
A,
Antenna
theory:
analysis
and

design.
John
wiley
&
sons,
2015.
<> “Welcome
to
Stable
Baselines
docs!
-
RL
Baselines
Made
Easy,”
https://stable-baselines.readthedocs.io,
accessed:
2021-10-19.

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO PARA. Downloaded on February 10,2022 at 17:42:35 UTC from IEEE Xplore. Restrictions apply.
View publication stats