IEEE INFOCOM 2021 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) | 978-1-6654-0443-3/21/$31.00 ©2021 IEEE | DOI: 10.1109/INFOCOMWKSHPS51825.2021.
9484529 IEEE INFOCOM WKSHPS: GI 2021: IEEE Global Internet Symposium
Resource Allocation for Low-Latency NOMA-V2X
Networks Using Reinforcement Learning
Huiyi Ding
Department of Electrical and Electronic Engineering
The University of Hong Kong
Pokfulam Road, Hong Kong, China
E-mail: hyding@eee.hku.hk
Abstract—With the development of the Internet of things
(IoT), vehicle-to-everything (V2X) plays an essential role in wire-
less communication networks. Vehicular communications meet
tremendous challenges in guaranteeing low-latency transmission
for safety-critical information due to dynamic channels caused
by high mobility. To handle the challenges, non-orthogonal
multiple access (NOMA) has been considered as a promising
candidate for future V2X networks. However, it is still an
open issue on how to organize multiple transmission links with
suitable resource allocation. In this paper, we investigate the
problem of the resource allocation for the low-latency NOMA-
integrated V2X (NOMA-V2X) communication networks. First,
a cross-layer optimization problem is formulated to consider
user scheduling and power allocation jointly while satisfying
the quality-of-service (QoS) requirements, including the delay
requirements, rate demands, and power constraints. To cope
with the limited time-varying channel information, a machine
learning based resource allocation algorithm is proposed to find
solutions. Specifically, reinforcement learning is applied to learn
the dynamic channel information for reducing the transmission
delay. The numerical results indicate that our proposed algo-
rithm can significantly reduce the system delay compared with
other methods while satisfying the QoS requirements, so as to
tackle the congestion issues for V2X communications.
Index Terms—Machine Learning, NOMA, Reinforcement
Learning, Resource Allocation, Vehicular Networks, V2X
I. I NTRODUCTION
Vehicle-to-everything (V2X) has attracted significant at-
tention due to the rapid development of vehicular tech-
nologies for beyond 5th generation (B5G) communications.
There have been greatly increasing vehicular applications,
including autonomous driving, advanced driver assistance, and
in-vehicle entertainment services. With the help of cellular
networks and device-to-device communications, cellular-V2X
(C-V2X) [1] enables diverse V2X communication modes si-
multaneously, including vehicle-to-vehicle (V2V), vehicle-to-
infrastructure (V2I), vehicle-to-pedestrian (V2P), and vehicle-
to-cloud (V2C).
To ensure improved quality of experience (QoE) of C-V2X,
B5G networks require ultra-reliable low-latency communica-
tions (URLLC). The resource allocation and mode selection
for V2X communication have been discussed in [2], where
the orthogonal multiple access (OMA) technique is used to
* Corresponding author.
978-1-6654-0443-3/21/$31.00 ©2021 IEEE
Authorized licensed use limited to: The University of Hong Kong Libraries. Downloaded on August 08,2021 at 00:50:28 UTC from IEEE Xplore. Restrictions apply.
Ka-Cheong Leung*
School of Computer Science and Technology
Harbin Institute of Technology, Shenzhen
Shenzhen 518055, China
E-mail: kcleung@ieee.org
assign different spectra to individual users. However, limited
radio resources cause significant competition among vehicles
and may degrade network performance. The stringent latency
and reliability requirements for V2X communications are
hard to achieve through the existing OMA technologies. The
successful realization of the URLLC entails the advent of new
technological concepts. Besides, the inter-user interference al-
ways exists in the OMA networks due to the carrier frequency
offset (CFO) caused by the Doppler effect of moving vehicles,
thereby limiting the system capacity.
To break through the restriction of limited frequency re-
source and reduce inter-user interference, non-orthogonal mul-
tiple access (NOMA) has been considered as a potential tech-
nique for vehicular networks, since it can support connections
with higher throughput, a lower latency, and higher reliability
than OMA [3]. In particular, NOMA applies superposition
coding (SC) at the transmitters for user multiplexing. It also
uses successive interference cancellation (SIC) at the receivers
to mitigate the inter-user interference. Hence, multiple users
can transmit concurrently with different power levels on the
same frequency domain in NOMA networks so as to avoid the
interference caused by the CFO effect. Resource allocation is
vital to improve the benefits of using NOMA. Specifically,
due to the multiplexed characteristics of SC and SIC, the
resource allocation for one user may affect the throughput
for other users in NOMA-integrated V2X (NOMA-V2X)
communication networks. Hence, there is a need to design a
suitable resource allocation method to fully utilize the benefits
of the NOMA technique.
Many existing studies are focusing on resource allocation
in NOMA networks, especially for optimization-based ap-
proaches. In [4], an interference-hypergraph-based resource
allocation algorithm has been devised, where the users with
smaller interference are multiplexed. However, the system
transmission rate is affected by the interference as well as the
channel gain. The proposed fixed power allocation limits the
performance gain of the NOMA-V2X networks. Therefore,
there is no guarantee on the system transmission rate for
the proposed heuristic method that has only considered the
minimization of the interference. The authors in [5] proposed
a power allocation scheme to maximize the system achievable
transmission rate while satisfying the predefined target thresh-
 IEEE INFOCOM WKSHPS: GI 2021: IEEE Global Internet Symposium
old of each NOMA user in the vehicular networks. However,
the approach does not consider the user scheduling order
so that it fails to satisfy variant quality-of-service quality-
of-service (QoS) requirements for different users. The mixed
centralized-distributed scheme was employed in [6] to formu-
late the resource allocation problem. However, the proposed
decentralized algorithm may cause severe interference with
greedy transmitters. A common feature of optimization-based
studies is the requirement of intensive computation to perform
the reallocation under time-varying network conditions, such
as fading and mobility. Hence, the aforementioned algorithms
may incur huge computational and communication overhead
during the re-allocation.
Besides, the above studies focus only on the physical
layer performance optimization, where the resource allocation
policy is adaptive to the channel information only. They do not
consider the stringent delay requirement for high-level packet
transmission. A typical assumption is that the data flow is
delay insensitive with an infinite backlog at the transmitter
side. In that case, the spectrum resource may be allocated to
one user with good physical channel condition, but there is no
packet to transmit at that period. Hence, resource allocation
under this assumption cannot accurately capture the behavior
of NOMA-V2X networks, leading to the wastage of the
spectrum resources and unsuccessful packet transmissions. It
is difficult to formulate the latency and reliability constraints
directly in the model-based optimization methods, since there
may not be closed-form expressions relating to the optimiza-
tion objective and constraints. Although some approaches
were proposed to deal with delay-aware resource allocation,
the latency was modeled as a long-term time-average delay
for the system in the optimization-based approach [7]. Those
approaches have to know the statistical information on channel
state and the arrival data rate, while this prior knowledge is
expensive to get or even unavailable.
To overcome this problem, machine learning (ML) has
attracted some attention, providing systems the ability to
automatically learn and improve through experience. It is
an efficient technique to solve the complex problem so as
to guarantee the latency and the reliability of the system
with dynamic channel state information (CSI). Due to the
absence of the perfect labeled data, reinforcement learning
(RL) becomes a potential approach for resource allocation in
NOMA-V2X communication networks. One of the advantages
of RL approaches is that they can operate efficiently without
relying on predefined models that characterize the environ-
mental statistics, which train the agent to do actions based
on the cumulative reward with the environmental feedback.
The authors in [8] applied RL to solve resource allocation
problems for vehicle platoon networks. However, only the
V2V mode is considered in that work with the number of
multiplexed vehicular users less than two, thereby under-
utilizing the spectrum resources in the system. In [9], RL is
applied to solve the resource allocation problems for vehicular
networks. Although two-user multiplexing is considered in
this study, the SIC technique has not been considered to
Authorized licensed use limited to: The University of Hong Kong Libraries. Downloaded on August 08,2021 at 00:50:28 UTC from IEEE Xplore. Restrictions apply.
Fig. 1. A NOMA-V2X System.
reduce the inter-user interference and enhance the benefits of
employing NOMA.
Motivated by the aforementioned difficulties, we investi-
gate the low-latency resource allocation for the NOMA-V2X
communication networks, aiming at the packet level delay
minimization. Specifically, We devise a scheme to consider
user scheduling and power allocation jointly, and formulate
it as a mixed-integer programming problem to minimize the
total system delay while satisfying other QoS requirements,
including the rate and the power constraints for all users.
In order to solve the complex problem efficiently, the RL
approach is applied to determine the user scheduling order,
which is then employed for power allocation by solving the
transformed feasible problem.
The paper is organized as follows. In Section II, we describe
the model of the system and formulate the optimization
problem. The proposed RL-based algorithm is discussed in
Section III. Performance evaluation is given in Section IV.
Finally, Section V gives the conclusion.
II. S YSTEM M ODEL AND P ROBLEM F ORMULATION
A. V2X Communications
As illustrated in Fig. 1, a typical V2X communication
system is considered in this work, consisting of one base
station (BS) and vehicular users. In the V2X applications,
vehicular communications refer to the case that messages
are transmitted directly to BS, such as updating the location
information or traffic data. Moreover, communication may
take place among different vehicles, such as the dissemination
of congestion messages. Following the typical representa-
tion, we classify the V2X communications into two main
types in our proposed scenario, namely, the V2I commu-
nication and V2V communication. In our model, one BS
with single antenna serves M uplink V2I users, denoted
by C = {1, 2, ..., Cm−1 , Cm , Cm+1 , ..., CM }. Besides, N
V2V links reuse the spectrum with V2I links, denoted by
D = {1, 2, ..., Dn−1 , Dn , Dn+1 , ..., DN }. Each V2V link
consists of one transmitter and one receiver. Each transmitter
or receiver is assumed to equip with a single antenna, which
can be extended to multiple antennas, similar to [10]. Every
transmitter maintains a buffer queue to store the packets to
 IEEE INFOCOM WKSHPS: GI 2021: IEEE Global Internet Symposium

be transmitted. Time is divided into equal-sized transmission The remaining interference to Cm caused by other V2I users
periods. The transmission packet load for vehicular user i is Bi at time slot k is:
in each transmission period. Due to the mobility of vehicles, X
semi-persistent scheduling (SPS) [11] is applied in the V2X ICC = XCi ,k PCi ,k HCi ,k (3)
Ci ∈C 0
communication networks. The resources are pre-allocated by
BS to the vehicles every transmission period. The system is where C 0 is the set of V2I users with HCi ,k < HCm ,k , and
assumed to operate in a certain transmission period with K PCi ,k denotes the transmit power for V2I user Ci at time slot
time slots, denoted by K = {1, 2, ..., k − 1, k, k + 1, ..., K}, k. The interference to Cm caused by other V2V links at time
where the time interval [k, k + 1) is referred to the time slot slot k is: X
k. At the beginning of each transmission period, BS decides IDC = XDi ,k PDi ,k HDi ,k (4)
the scheduling order and responses for the power allocation. Di ∈D 0
Actually, the scheduling order affects system throughput and 0
where D is the set of V2V links with HDi ,k < HCm ,k , and
delay. Scheduling all users in the same time slot induces PDi ,k denotes the transmit power for V2V link Di at time slot
serious inter-user interference or even impractical for the k. Thus, the received signal-to-interference-plus-noise ratio
hardware design of the SIC technique. Besides, the system (SINR) for V2I users Cm at time slot k can be expressed
transmission rate is affected by power allocation. Hence, user as:
scheduling and power allocation schemes are needed to be XCm ,k PCm ,k HCm ,k
ΓCm ,k = (5)
carefully designed. ICC + IDC + N0
where PCm ,k and N0 denote the transmit power for V2I user
B. NOMA-V2X Networks Cm and the Gaussian noise at time slot k, respectively.
NOMA is a promising technique to improve the spectrum Similarly, the SINR for V2V link Dn at time slot k is
efficiency for wireless networks. By applying NOMA, multi- written as:
XDn,k PDn,k HDn,k
ple users can transmit data simultaneously on the same spec- ΓDn,k = (6)
trum with different power levels. In particular, SIC is applied ICD + IDD + N0
at the receivers to remove the partial inter-user interference. where PDn ,k denotes the transmit power for V2V link Dn at
As indicated in [12], the optimal decoding in the NOMA time slot k. The interference to Dn caused by other V2I users
transmission is the descending order in the channel gain to at time slot k is:
achieve the maximum system capacity. Specifically, in order X
ICD = XCi,k PCi,k HCi,k (7)
to decode the own signal of each user at the receiver side, the
Ci ∈C 00
signals from other users with the stronger channel gain are
00
decoded first and removed from the combined signals, while where C is the set of V2I users with HCi ,k < HDn ,k . The
the remaining signals from users with smaller channel gain interference to Dn caused by other V2V links at time slot k
are regarded as interference. is: X
In order to improve the reliability and reduce the delay of IDD = XDi,k PDi,k HDi,k (8)
the systems, NOMA is applied in the V2X communication Di ∈D 00
networks. In the proposed NOMA-V2X system model, the 00
where D is the set of V2V links with HDi ,k < HDn ,k .
channel between each transmitter and receiver suffers from According to the Shannon-Hartley theorem [13], the achiev-
Rayleigh fading and path loss with time-varying CSI. The able rate obtained from V2I user Cm and V2V link Dn at time
channel coefficient Hi,k for link i in time slot k is defined as: slot k can be calculated as:
Hi,k = |Gi,k (Li,k )−z |2 (1) RCm,k = log2 (1 + ΓCm,k ) (9)

where Gi,k and z denote the Rayleigh fading and path loss RDn,k = log2 (1 + ΓDn,k ) (10)
exponent between transceivers for each link, respectively. The Thus, the total system transmission rate in one transmission
distance between each transmitter and receiver is defined as: period can be expressed as:
Li,k = di,k + vi,k Tia
X X X X
(2) Rtot = RCm,k + RDn,k (11)
Cm∈C k∈K Dn∈D k∈K
where di,k , vi,k , and Tia denote the initial distance at the start
of a transmission period, the velocity, and the access delay The delay for user i is defined as Ti = Tia + Tit , where the
a
P
for user i, respectively. access delay is: Ti = k∈K kXi,k , and the transmission
At each transmission period, a binary indicator, denoted as delay is denoted as: Tit = RBi,k i
, where Bi is the size of
Xi,k , is the scheduling policy decided by BS for link i in each transmission load. Therefore, the total system delay in one
time slot k. In particular, Xi,k = 1 when the link is scheduled, transmission period is given as:
otherwise, Xi,k = 0. After applying NOMA, part of the inter- Ttot =
X
TCm +
X
TDn (12)
user interference can be canceled by using the SIC technique. Cm∈C Dn∈D

Authorized licensed use limited to: The University of Hong Kong Libraries. Downloaded on August 08,2021 at 00:50:28 UTC from IEEE Xplore. Restrictions apply.
 IEEE INFOCOM WKSHPS: GI 2021: IEEE Global Internet Symposium
C. Problem Formulation
Compared with traditional cellular communications, the
newly proposed C-V2X applications, such as sensor mon-
itoring and autonomous vehicle platooning, require tighter
latency and higher reliability [14]. Considering the stringent
requirements in NOMA-V2X communication networks, we
aim to propose an algorithm to minimize the system latency
while satisfying the reliability requirements. In particular, the
reliability requirements can be expressed as the transmission
rate and power constraints. The transmission rate larger than
the threshold with an acceptable transmission power guaran-
tees the successful transmission for each user. According to
(11) and (12), system performance can be determined based
on user scheduling order and power allocation. Therefore, we
formulate a joint user scheduling and power allocation scheme
as an optimization problem, which can be expressed as:
min Ttot
X,P
s.t. C1 : 0 ≤ Pi ≤ Pimax , i ∈ C ∪ D
th
C2 : RPi ≥ Ri , i∈C∪D
C3 : Xi,k = 1, i ∈ C ∪ D
k∈K
Xi,k ≤ X max , k ∈ K
P
C4 :
i∈C∪D
where Pimax and Rithdenote the maximum transmit power
and the rate threshold of successful transmission, respectively.
In (13), C1 limits the peak transmit power for each users. C2
maintains the successful transmission for each users. C3 and
C4 are the binary constraints for the user scheduling variables
so as to make sure that each user can be scheduled during a
transmission period, while the total number of multiplexed
users is satisfied for the SIC hardware design.
III. RL- BASED R ESOURCE A LLOCATION S CHEME
The aforementioned optimization problem (13) is an NP-
hard mixed-integer programming problem, which is difficult
to be solved by the conventional optimization-based meth-
ods. Although some algorithms, such as the branch-and-
bound method, can approximately solve the problem, the
computational complexity is still expensive. Besides, such
optimization-based algorithms can only solve the problem
with the known CSI information for K time slots. However,
that information cannot be accurately predicted in real sit-
uations. In order to overcome these challenges, ML can be
applied to find solutions for resource allocation. Specifically,
RL is a suitable candidate to deal with imperfect information,
which can interact with a dynamic channel environment,
and learn both the behaviors of users and the environmental
conditions online so as to maximize the system utility.
Hence, we reformulate the optimization problem as an RL
task by four tuples as φ = {a, s, r, π}, where a, s, r and
π denote the actions space, state space, reward, and action
policy, respectively. In state s(t), the action a(t) taken by
the agent is choosing by the policy π based on the collective
information of the environment. The reward r(t) is obtained
by taking the specific action, and the environment evolves into
Authorized licensed use limited to: The University of Hong Kong Libraries. Downloaded on August 08,2021 at 00:50:28 UTC from IEEE Xplore. Restrictions apply.
the next state s(t + 1). The specific settings are discussed in
the following:
1) State: In this work, the BS is treated as the agent.
The state of the environment is characterized by the CSI of
the NOMA-V2X communication network. The environment
state for link i at episode t is thus defined by si (t) =
[Hi,1 (t), ..., Hi,K (t)] ∈ s, which reflects the environment dy-
namics. The size of state space is denoted as Ns . Specifically,
we assume that a BS has limited knowledge about imperfect
CSI with path loss information, due to the mobility of vehicles
in the NOMA-V2X communication networks.
2) Action: Due to the unpredictable environmental
changes, BS needs to take the appropriate action for the user
scheduling and power allocation based on the observed chan-
nel information. Based on (13), the optimal power allocation
can be conducted by a convex problem with the obtained user
scheduling order:
min Ttot
P
(14)
s.t. C1 : 0 ≤ Pi ≤ Pimax , i ∈ C ∪ D
(13) C2 : Ri ≥ Rith , i ∈ C ∪ D
The convexity can be proved by following a similar way in
[15] and the proof is omitted here because of constraints
in space. Thus, instead of the action space formulation,
we convert it into the penalty for ensuring that the QoS
requirements of the system can be satisfied. To deal with
the discrete problem for user scheduling, we apply the Q-
learning framework to learn the NOMA-V2X communication
networks. Q-learning seeks to find the best actions from the
action space with given state information by creating and
updating a Q-table. The action for each link i at episode t
is denoted as ai (t) = [Xi,1 (t), ..., Xi,K (t)] ∈ a, which is
chosen according to the -greedy policy π as:
a(t) = arg max Q(s(t), a0 (t)) , with probability 
(
a0 (t)
π:
a(t) = random action a0 (t) , with probability 1 − 
(15)
Specifically, the -greedy policy here takes the greedy action
a(t) to maximize the Q-value with probability , which
guarantees the performance of the policy. Besides, the random
action is chosen with probability 1 −  to ensure the full
exploration for the problem for finding the global optimal
policy.
3) Reward: The reward is regarded as the feedback from
the environment. The agent has a higher probability of se-
lecting an action that brings a larger reward value. In this
work, the aim for BS consists of the delay reduction and the
transmission rate increment. According to the characteristics
of the system model (13), the reward function at episode t is
defined as:
X
r(t) = −Ttot (t) + c · I((Ri (t) − Rith (t))) (16)
i∈C∪D
where the first term is a utility function to minimize the
delay, and the second term is penalty part, which ensures the
requirements on the system transmission rates, where I(·) is
 IEEE INFOCOM WKSHPS: GI 2021: IEEE Global Internet Symposium
a sign function and it equals one when the rate requirement
is satisfied. After obtaining the reward, the Q-table in the Q-
Algorithm 1 RL-RA
1: Initialize State space, Action space, Q-table
2: Set Emax , , α, and γ
3: for each episode e < Emax do
4: if In random chooss policy then
5: Randomly choose action X
6: end if
7: if In -greedy policy then
8: Choose action X with maximum Q-value
9: end if
10: Get P by solving the convex problem (16)
11: Observes the state transition; update Q-table
12: end for
13: Output: X and P
learning model will be updated by:
Q(s(t), a(t)) ←Q(s(t), a(t))(1 − α)
+ α[r(t) + γ max
0
Q(s(t + 1), a0 (t))] (17)
a (t)
where α and γ denote the learning rate and the reward
decay, respectively. The details for the proposed RL-based
resource allocation algorithm, denoted as RL-RA, is shown
in Algorithm 1. Specifically, Step 1 initializes the variables
for the NOMA-V2X network and the RL framework. From
Steps 4-9, the action is taken by BS for user scheduling
following the policy π. Step 10 shows the power allocation
approach. The updating of environmental variables is illus-
trated in Step 11. The computational complexity of RL-RA is
O((N + M )KNe ), where Ne denotes the value of episodes.
To show the convergence of the algorithm, we prove that the
following relation:
A B
f (X∗ , P∗ ) ≥ f (X, P∗ ) ≥ f (X, P)
∗ ∗
always holds, where f , X , and P denote the original
optimization problem, the optimal user scheduling, and the
optimal power allocation, respectively. The inequation (A)
follows the convergence results on RL approaches [16], while
inequation (B) holds from the definition of the convex opti-
mization [17]. We, therefore, conclude that the solution found
by Algorithm 1 converges since the difference in values
between the calculated solution and the true optimal one is
bounded.
IV. P ERFORMANCE E VALUATION
In this section, we illustrate the performance of the pro-
posed algorithm using Python 3.6. We consider a NOMA-
V2X communication network with one BS located at the cell
center. The cell radius can be set as 50 meters to fulfill the
requirement of vehicular communications, similar to [14]. The
path loss component and the noise are set as three and −174
dBm/Hz. We set the V2I and V2V links as four and six,
respectively, and the maximum speed for each user can be
Authorized licensed use limited to: The University of Hong Kong Libraries. Downloaded on August 08,2021 at 00:50:28 UTC from IEEE Xplore. Restrictions apply.
Delay vesus episodes
30
RL-RA
MAX-RATE
25 RANDOM-US
20
Total Delay (ms)
15
10
5
0
0 20 40 60 80 100
Episodes
Fig. 2. Total system delay comparison among various algorithms.
60 km/h, which follows [5]. We set the maximum transmit
powers for V2I and V2V users are set as 4 watts and 2
watts, respectively. As far as the Q-learning parameters are
concerned, the reward decay γ, the learning rate α, and
the greedy probability coefficient  are set as 0.5, 0.1, and
0.95, respectively. We compare the total transmission delay
of our proposed algorithm (RL-RA) with two commonly
used baseline resource allocation algorithms, namely: A1)
the heuristic algorithm proposed in [5], which focuses on
the power allocation with random user scheduling, denoted
as RANDOM-US; and A2) the method raised in [6], which
concentrates on the physical layer optimization in order to
maximize the system transmission rate, denoted MAX-RATE,
respectively.
Fig. 2 illustrates the achieved total system delay of dif-
ferent algorithms. It can be seen that our proposed RL-RA
algorithm outperforms both the MAX-RATE and RANDOM-
US methods with a smaller total delay. The performance
(18) of the RL-RA algorithm is improved and finally converged
as the episode increases. The total system delay for the
MAX-RATE algorithm is even larger than RANDOM-US,
since the MAX-RATE algorithm only considers the physical
layer optimization for the system transmission rate. Hence,
choosing a smaller number of multiplexed users can reduce
the inference in order to maximize the transmission rate,
thereby causing a larger system delay. Besides, some spectrum
resources may be allocated to the users without packets being
sent. Therefore, the trade-off between system throughput and
total delay should be considered carefully in the resource
allocation scheme.
Fig. 3 shows the total system delay with various con-
figurations on the number of V2V users in the system for
our proposed RL-RA algorithm. It is observed that the total
system delay is reduced with increasing episodes and finally
converged for all the cases in the RL-RA algorithm. Besides,
the total system delay for the two-V2V system is less than
the four-V2V system and six-V2V system. It shows that
the number of total users affects the user scheduling order.
 IEEE INFOCOM WKSHPS: GI 2021: IEEE Global Internet Symposium

tion scheme. In order to deal with the complex cross-layer

Delay for NOMA-V2X with RL problem with dynamic channel information, we propose an
Two-V2V
Four-V2V
RL-based resource allocation algorithm to jointly consider the
20 Six-V2V user scheduling and power allocation schemes. Performance
evaluation is conducted to demonstrate that the proposed RL-
15 RA algorithm can significantly reduce the total system delay
Total Delay (ms)

while satisfying rate and power QoS requirements, compared

with other baseline methods. In the future, we plan to extend
10
our work to the multi-cell systems to analyze the complex
scenarios among vehicles across cellular cells.
5
R EFERENCES
[1] M. Gonzalez-Martı́n, M. Sepulcre, R. Molina-Masegosa, and J. Goza-
0 lvez, “Analytical Models of the Performance of C-V2X Mode 4 Ve-
0 20 40 60 80 100
Episodes hicular Communications,” IEEE Transactions on Vehicular Technology,
vol. 68, no. 2, pp. 1155–1166, Dec. 2019.
[2] X. Li, L. Ma, R. Shankaran, Y. Xu, and M. A. Orgun, “Joint Power
Fig. 3. Total system delay with different number of users. Control and Resource Allocation Mode Selection for Safety-related
V2X Communication,” IEEE Transactions on Vehicular Technology,
vol. 68, no. 8, pp. 7970–7986, Jun. 2019.
[3] D. Zhang, Y. Liu, L. Dai, A. K. Bashir, A. Nallanathan, and B. Shim,
Delay for NOMA-V2X with RL
“Performance Analysis of Decentralized V2X System with FD-NOMA,”
25 EPSILON = 0.85
in 2019 IEEE 90th Vehicular Technology Conference, Sep. 2019.
EPSILON = 0.9
EPSILON = 0.95
[4] C. Chen, B. Wang, and R. Zhang, “Interference Hypergraph-Based
Resource Allocation (IHG-RA) for NOMA-Integrated V2X Networks,”
20
IEEE Internet of Things Journal, vol. 6, no. 1, pp. 161–170, Oct. 2019.
[5] H. Xiao, Y. Chen, S. Ouyang, and A. T. Chronopoulos, “Power Control
Total Delay (ms)

15
10
5 vol. 35, no. 10, pp. 2383–2397, Jul. 2017.
0 20 40 60
Episodes
Fig. 4. Total system delay with different learning settings.
Besides, it can be seen that the convergence speed is faster
for the system with a smaller number of V2V users. Indeed,
the system with a smaller number of V2V users has a smaller
action space, thereby a faster convergence.
We further investigate the impact of greedy probability
coefficient  in the learning setting of RL-RA algorithm. As
shown in Fig. 4, it can be observed that the converge speed
increases with the increment of . Actually,  relates to the
action choice. A larger value of  means that there is a large
probability to choose a greedy action with the maximum
reward. Otherwise, a random action will be chosen to explore
the environment. Actually, the greedy action ensures the
performance of the policy based on the known environment,
and the random action encourages the state space exploration
for the unknown environment.
V. C ONCLUSION
In this work, we study a low-latency NOMA-V2X network
to support both V2I and V2V communications. To deal
with the delay requirements, we formulate the mixed integer
programming optimization problem for the resource alloca-
for Clustering Car-Following V2X Communication System With Non-
Orthogonal Multiple Access,” IEEE Access, vol. 7, pp. 68160–68171,
May 2019.
[6] B. Di, L. Song, Y. Li, and G. Y. Li, “Non-Orthogonal Multiple
Access for High-Reliable and Low-Latency V2X Communications in
5G Systems,” IEEE Journal on Selected Areas in Communications,
[7] H. Ding and K.-C. Leung, “Cross-Layer Resource Allocation in NOMA
Systems with Dynamic Traffic Arrivals,” in 2020 IEEE Wireless Com-
80 100 munications and Networking Conference (WCNC), May 2020.
[8] S. Xu, C. Guo, and Z. Li, “NOMA Enabled Resource Allocation for
Vehicle Platoon-Based Vehicular Networks,” in 2019 IEEE Globecom
Workshops (GC Wkshps), Dec. 2019.
[9] Y. Xu, C. Yang, M. Hua, and W. Zhou, “Deep Deterministic Policy Gra-
dient (DDPG)-Based Resource Allocation Scheme for NOMA Vehicular
Communications,” IEEE Access, vol. 8, pp. 18797–18807, Jan. 2020.
[10] J. Ding and J. Cai, “Two-Side Coalitional Matching Approach for
Joint MIMO-NOMA Clustering and BS Selection in Multi-Cell MIMO-
NOMA Systems,” IEEE Transactions on Wireless Communications,
vol. 19, no. 3, pp. 2006–2021, 2020.
[11] Third Generation Partnership Project (3GPP), “Technical Specification
Group Radio Access Network,” Study on LTE-Based V2X Services,
3GPP TR 36.885 V0.5.0, Release 14, Feb. 2016.
[12] L. Dai, B. Wang, Y. Yuan, S. Han, C. I, and Z. Wang, “Non-orthogonal
Multiple Access for 5G: Solutions, Challenges, Opportunities, and
Future Research Trends,” IEEE Communications Magazine, vol. 53,
no. 9, pp. 74–81, Sep. 2015.
[13] J. R. Pierce, “An Introduction to Information Theory: Symbols, Signals
and Noise,” Courier Corporation, vol. 2, Nov. 1980.
[14] M. Patra, R. Thakur, and C. S. R. Murthy, “Improving Delay and
Energy Efficiency of Vehicular Networks Using Mobile Femto Access
Points,” IEEE Transactions on Vehicular Technology, vol. 66, no. 2,
pp. 1496–1505, May 2017.
[15] Z. Yang, W. Xu, C. Pan, Y. Pan, and M. Chen, “On the Optimality of
Power Allocation for NOMA Downlinks With Individual QoS Con-
straints,” IEEE Communications Letters, vol. 21, no. 7, pp. 1649–1652,
Mar. 2017.
[16] S. Singh, T. Jaakkola, and M. L. Littman, “Convergence Results for
Single-Step On-Policy Reinforcement-Learning Algorithms,” Machine
Learning, vol. 38, no. 3, pp. 287–308, Mar. 2000.
[17] K. Wang, W. Liang, Y. Yuan, Y. Liu, Z. Ma, and Z. Ding, “User
Clustering and Power Allocation for Hybrid Non-Orthogonal Multiple
Access Systems,” IEEE Transactions on Vehicular Technology, vol. 68,
no. 12, pp. 12052–12065, Oct. 2019.

Authorized licensed use limited to: The University of Hong Kong Libraries. Downloaded on August 08,2021 at 00:50:28 UTC from IEEE Xplore. Restrictions apply.