IEEE COMMUNICATIONS LETTERS, VOL. 24, NO.
7, JULY 2020
Cooperative Spectrum Sensing Meets Machine Learning:
Deep Reinforcement Learning Approach
Rahil Sarikhani
Abstract— Cognitive radio network (CRN) emerged to utilize
the frequency bands efficiently. To use the frequency bands
efficiently without any interference on the licensed user, detection
of the frequency holes is the first step, which is called spectrum
sensing in the context. In order to increase the quality of local
spectrum sensing results, cooperative spectrum sensing (CSS) is
introduced in the literature to combine the local sensing results.
Recently, machine learning techniques are designed to improve
the classification of the images and signals. Specifically, Deep
Reinforcement Learning (DRL) is of interest for its substantial
improvement in the classification problems. In this letter, we have
proposed DRL based CSS algorithm, which is employed to
decrease the signaling in the network of SUs. The simulation
results represent the superiority of the proposed approach to
state-of-the-art approaches, including Deep Cooperative Sensing
(DCS), K-out-of-N, and Support Vector Machine (SVM) based
CSS algorithms.
Index Terms— Cognitive radio, cooperative spectrum sensing,
deep cooperative sensing, deep reinforcement learning, machine
learning.
I. I NTRODUCTION
C OGNITIVE radio networks (CRN) are exceptional cases
of networks that have been designed to utilize the
frequency bands dynamically and efficiently. Two crucial
players of these networks are primary users (PU) who pay
for the frequency bands and secondary users (SU), which
utilize the vacant bands dynamically. Hence, determining the
unoccupied bands is crucial for the SUs due to not interrupt
the PUs. Although the spectrum sensing is essential in each
SU, the individual sensing results are highly disposed to the
fading channel conditions and other destructive effects [1]–[3].
Consequently, to improve the sensing quality, Cooperative
Spectrum Sensing (CSS) is introduced in the literature. CSS
utilizes the results of some of cooperating SUs to enhance
the quality of sensing and develop the accuracy. However,
the channel condition affects the optimal strategy of coop-
eration in CSS [1], [2].
Accordingly, some efficient approaches based on K-out-of-N
scheme have been introduced [1]. Moreover, multi-
dimensional correlation in individual sensing was considered
for the CSS in [2]. Besides, machine learning-based
approaches were introduced in [3] and [4], which improve
Manuscript received February 4, 2020; revised March 5, 2020; accepted
March 19, 2020. Date of publication March 30, 2020; date of current version
July 10, 2020. The associate editor coordinating the review of this letter
and approving it for publication was M. Chafii. (Corresponding author:
Rahil Sarikhani.)
Rahil Sarikhani is with the Department of Computer Engineering,
Kerman Branch, Islamic Azad University, Kerman 7635131167, Iran (e-mail:
rahil.sarikhani@iauk.ac.ir).
Farshid Keynia is with the Department of Energy Management and Opti-
mization, Institute of Science and High Technology and Environmental
Sciences, Graduate University of Advanced Technology, Kerman 7635131167,
Iran.
Digital Object Identifier 10.1109/LCOMM.2020.2984430
1558-2558 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY WARANGAL. Downloaded on November 30,2022 at 14:20:54 UTC from IEEE Xplore. Restrictions apply.
1459
and Farshid Keynia
the quality of the well-known procedures in the fusion
center (FC). Recently, reinforcement learning (RL) and deep
learning (DL) are combined to improve the accuracy of
machine learning [5], [6].
In this letter, we have utilized deep reinforcement learning
(DRL) to improve the classification performance of CSS. The
DL-based CSS needs the updating of the measurement from
all the SUs in the network. While it would be very resource-
consuming and network flow consuming. In the proposed
approach, we have employed Reinforcement Learning (RL) to
introduce the required SU to update its measurements. Hence,
the proposed DRL can increase the CSS efficiently from a
resource and time viewpoint. Initially, the agent selects an
SU to share its local sensing based on some approaches, and
then the selected SU is notified. Subsequently, the selected
SU determines the presence of the PU based on local energy
detection and acquaint the agent from its local sensed energy
values. Then, the agent determines the presence or absence
of the PU, globally. Finally, the Q-learning parameters are
updated based on the sensing results, and this approach is
continued to the convergence of the algorithm. Accordingly,
we have formulated the CSS problem as a Markov Decision
Problem (MDP) and introduced state and action spaces and
strategy selection policy. Moreover, the reward function is
introduced based on the correlation of cooperating users.
Furthermore, we have defined a DL unit to be employed by the
proposed DRL algorithm in order to increase the reliability in
large scale problems. Utilizing the proposed DRL approach
decreases the network flow and the number of cooperat-
ing users, which yield in less resource-consuming approach
for CSS. Besides, the proposed approach represents a more
robust criterion because of omitting correlated measurements.
The remainder of the letter is as follows: After the system
model in Section II, the DRL based CSS algorithm is consid-
ered in Section III. Subsequently, the simulation results and
concluding remarks are expressed in Section IV and V.
II. S YSTEM M ODEL
We consider a CRN included NSU SU users who try to
accompany sensing the spectrum holes to transmit the in-queue
data. The positions of the SUs are randomly selected in each
scenario. Moreover, some PUs are located randomly in the
network and working in the licensed network. Furthermore,
we consider a wideband channel to be sensed with NB bands
and the total bandwidth of W .
The PUs can transmit in NP bands arbitrarily, and none
of the NSU are aware of the utilized bands. The power of
transmitting is fixed at P . Additionally, the Additive White
Gaussian Noise (AWGN) is assumed where the power spectral
density is N0 and zbi (n) is the noise of the i-th SU in the
band b at the time of n. Moreover, the fading and shadowing
 1460
effect, together with the hidden PU, makes some SUs unable to
detect the presence of the PUs by local sensing. Hence, the SU
broadcasts the request of cooperative sensing, and all the one-
hop neighbors will respond to its appeal by their local sensing
results, intermittently. The initiating SU, which requests for
cooperative sensing, is called the agent in this phase. The
agent will combine the local sensing results and take the role
of Fusion Center (FC) in the network.
The FC, which is called the agent from now on in this
letter, begins the DRL-based cooperative spectrum sensing
and detects the presence or absence of the PU based on
the surrounding one-hop neighbors. The number of one-hop
neighbors surrounding the agent is N . In each period of
cooperative sensing, C < N nodes are selected based on the
reliance of the cooperation, which is estimated by the location-
based correlation metric ρji , and it would change based on the
conditions of the channel and location of SUs. These C nodes
are informed step-by-step to update their local sensing results.
To broadcast the fusion decision, the agent utilizes a reliable
channel to transmit its cooperation consequence and inform
other one-hop nodes about the last result. Besides, the channel
is assumed to be unavailable if all of the available bands
are detected to be occupied by the PUs. The same as other
references, such as [3], the birth-death process is assumed for
the simulation of the PU activity.
Let yi,j be the received energy samples from the SU i from
the j-th band as follows
1 
N
yi,j = |xji (n)|2
NED n=1
where xji (n) is the nth received sample in the jth band of
ith SU. Moreover, NED is the number of samples to be
employed for energy detection. The agent collects all the
measurements in the matrix Y = {yi,j } ∈ RC×NB . After
collecting all the measurements, the agent will determine the
value function using DL select an appropriate action from
action space A. Then, it receives the local sensing values
and updates the measurement matrix. By running the NN,
it calculates the new Q values and selects the other node
(appropriate action) if needed. At the end of cooperation,
it will inform all the users from the cooperated result.
III. D EEP R EINFORCEMENT L EARNING -BASED
C OOPERATIVE S PECTRUM S ENSING
The agent takes action from A in each episode of coop-
eration, and the environment will be affected by the selected
action. This influence will be replied by the environment uti-
lizing two main phenomena called the state and reward. Then,
the reward will be used in the agent, and the value function
will be determined using deep learning. In the following, all
the parameters in the learning process will be determined.
A. Reinforcement Learning
In this part, we will explain the sets of actions A, states S,
and reward calculation methodology in order to determine
the RL based wide-band spectrum sensing. The state set S
is involved in S different states, which are every other node
together with two additional states, including start and stop
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY WARANGAL. Downloaded on November 30,2022 at 14:20:54 UTC from IEEE Xplore. Restrictions apply.
IEEE COMMUNICATIONS LETTERS, VOL. 24, NO. 7, JULY 2020
states. Consequently, if we represent each state by si for
i = 0, 1, . . . , NSU + 1, then state s0 is the start state and
sNSU +1 is the stop state and others are according to the SUs
in the spectrum sensing network. Subsequently, the actions in
each state are selected from the set of actions as A, which can
be determined by a0 , a1 , . . . , aA . In this set, a0 is according to
the final action, which means the learning is finalized. Others
are identical to the secondary users in the spectrum sensing
network. It means when an agent selects the action ak for
k = 0, the secondary user nk is selected to transmit its data
about the spectrum. Action selection in each agent is based
on Boltzman distribution. In other words, the probability of
action ak is based on Boltzman distribution which can be
formulated as
eQ(sk ,ak =i)/τk
p(ak = j|sk ) = NSU (2)
Q(sk ,ai )/τk
i=1 e
for j = 1, . . . , NSU . In this equation, Q(sk , ak ) is the state-
action value function, which determines the value of selecting
action ak in state sk and ςk is the temperature control which
balances the exploration versus exploitation in each stage
of the algorithm. In this algorithm, increasing the value of
the temperature will increase the independence to the large
value of Q(sk , ak ). Consequently, the agent will explore more
opportunities. Since the notion of the algorithm should vary
from exploration to exploitation, the value of the temperature
is chosen to be linearly decreased in each stage to improve
the exploitation versus exploration.
(1) For the reward calculation, the correlation of the newly
selected SU is considered. Since each SU is located in different
places in the cognitive network, the sense of the PU will be
correlated or uncorrelated between them. It can be enumerated
by the correlation coefficient which is calculated by location
information of the SUs [7]. Based on this information, the cost
is calculated by the summation of the correlation coefficients
from the first step up to now. Accordingly, the correlation of
the kth-step of cooperation could be calculated as

k−1
Ck = ρji (sm , am = j) (3)
m=0
As it is known, the best policy which is the goal of RL
algorithm is thepolicy π in which the discounted reward
function Rt = t=0 γ rt for discount factor 0 < γ ≤ 1
t
is maximized. Here, we utilized a neural network in the
context of deep Q-network to estimate the optimal action-value
function (Q) as:
Q∗ (s, a) = max E(rt + γrt+1 + . . . |st = s, at = a, π) (4)
π
This function represents the maximum of the expected dis-
counted reward under policy π which takes action a in state s.
A simpler form of this equation is considered using Bellman
Equation.
 
Q∗ (s, a) = Es r + γ max
Q∗ (s , a )|s, a (5)
a
For the known value Q (s , a ), where s and a are the next
∗  
states in the sequence of state action spaces. The optimal
policy is the action a which maximizes the expected value
of r + γQ∗ (s , a ). A nonlinear function estimator is used
 SARIKHANI AND KEYNIA: CSS MEETS MACHINE LEARNING: DRL APPROACH 1461

to approximate the action-value function in deep Q-network. Algorithm 1 DRL-Based Cooperative Spectrum Sensing
If the weights of the utilized neural network are assumed to 1: Input: A, N, Q
be θ, then Q(s, a; θ) ≈ Q∗ (s, a). In such networks, the training 2: Output: K  , P 
is based on the minimizing of the updated loss function Li (θi ). 3: for n = 1 to N do
To integrate the RL and DL, the data-set of the recently 4: for k = 0 to A + 1 do
experienced transition and experience replay, which represents 5: ak is selected according to Eq. (2).
the recent experiences, are critical. The loss function, which is 6: if ak = 0 then
used in each iteration i in the Q-learning update is as follows: 7: Request of channel sensing is sent to the SUak
 
Li (θi ) = E (r + γ max Q(s  
, a ; θ ) − Q(s, a; θ ))2
(6) 8: The energy of the sub-channels is sensed by the

i−1 i
a selected SU
where θi and θi−1 are the current and previous network 9: for i = 1 to 3 do
parameters. Accordingly, the gradient of the loss function 10: Convolutional Layer,
could be formulated as 11: ReLU,
 12: Max pooling Layer,
∇θi Li (θi ) = E (r + γ max

Q(s , a ; θi−1 ) − Q(s, a; θi )) 13: end for
a
×∇θi Q(s, a; θi )] (7) 14: Fully Connected,
15: ReLU,
16: Fully Connected,
B. Deep Learning
17: rk+1 (sk , ak ) = C1k based on Eq. (3)
In the agent, the Convolutional Neural Network (CNN) 18: the parameters of learning are updated,
model is utilized to combine individual sensing consequences 19: else
to calculate the action-value function for the DRL to determine 20: break,
the absence or presence of the PU. The proposed CNN for 21: end if
calculating the action-value function is composed of the con- 22: end for
volution part and fully connected (FC) part as two main parts. 23: u = CoopSenDec(ui , k, A)
The input is the matrix Y which is included of the energy 24: Broadcast the sensing result
samples from neighbor nodes in different bands. The Y matrix 25: end for
is entered into the convolution part, which composed of three
different sub-blocks called convolutional layer, rectifier linear
unit (ReLU), and max-pooling layer where connected in Indian mean that the agent decides the presence or absence of the PU
file. The details of each sub-block can be inferred from [5], based on the local results.
[6]. Specifically, ReLU chooses the maximum value among
the input and zero. Moreover, the max-pooling unit, which is IV. N UMERICAL R ESULTS
fed by a vector, selects the maximum entry of the vector. Three Here, we will evaluate the performance of the proposed
consecutive sub-blocks are included in each convolution layer. algorithm to the other machine learning based approaches. All
Subsequently, two FC sub-blocks are included. FC sub-blocks the simulations are done on MATLAB software. A single PU
are responsible for the gathering of features extracted from the is assumed which is located randomly among number of SUs
convolutional layers. To this end, these two sub-blocks mix the in an area limited by 500 meters in 2 dimensions. Furthermore,
extracted features utilizing the weighting process and biasing the number of frequency bands is set to NB = 160 where each
process. Furthermore, an extra ReLU is positioned between sub-band occupies 1 MHz of bandwidth. Moreover, the trans-
two FC sub-blocks to handle the non-linear classifications. The mitter power is assumed to be P = 30 dBm, and β = 3000,
details can be obtained from [5], [6]. α = 3.8, and σ = 7.9 dB [8], where β, α, and σ denote path
loss constant, path loss exponent, and the standard deviation
C. Deep Reinforcement Learning of the shadow fading, respectively. Besides, the temperature
The DRL-based cooperative spectrum sensing algorithm is control is initialized by 0.01 and increased exponentially in the
represented in Algorithm 1 step-by-step. In this algorithm, iterations. Furthermore, the training of the required coefficients
the number SUs, and all the states together with the actions are in DL is performed based on adaptive moment estimation
considered as input. The output would be the result of CSS. algorithm. Additionally, spatially correlated fading is assumed
In the agent, one of the available SUs is selected and informed for spatial correlation sensing which is modeled based on
dA−B
( )
to represent its local sensing result. The local sensing results ρA−B = e dref with dref = 50 m [9].
are gathered by measuring the available energy in different To evaluate the performance of the proposed algorithm,
subchannels and transmitted to the agent. After that, the algo- we have compared the proposed approach with other machine
rithm begins to run in the agent. To this end, the sensing learning-based cooperative spectrum sensing, including K-out-
matrix Y is gathered, and after three convolutional layers and of-N, SVM [3], and DL [4] approaches as the state-of-the-art
two subsequent FC layers, the result of global sensing through schemes. Furthermore, the comparison metrics are Region of
CNN is obtained. Then, in the agent, the action-value and Convergence (ROC), number of active SUs, and the sensing
rewards are updated according to the cost function and the error, which is the average of probability of false alarm (Pf )
result of local sensing. In the algorithm, by CoopSenDec we and the probability of the miss detection (Pm ).

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY WARANGAL. Downloaded on November 30,2022 at 14:20:54 UTC from IEEE Xplore. Restrictions apply.
 1462
Fig. 1. The ROC of the proposed scheme for P = 30 dBm.
Fig. 2. Number of active users vs. number of cooperating SUs.
In Fig. 1 the ROC, which is the probability of detection
(Pd ) in corresponding Pf of four mentioned schemes are
compared. Apparently, the proposed approach outperforms
others in a lower Pf , while in a higher ones, the result of SVM
approach is comparable to the proposed approach. Of course,
in lower Pf , the results of the proposed approach is compara-
ble with the result of the DL-based sensing method. Moreover,
the area under the curve of these three schemes is calculated as
0.822, 0.943, 0.958, and 0.961 for K-out-of-N, SVM, DL, and
proposed schemes, respectively. Obviously, the area under the
curve of the schemes represents the highest sensing accuracy
than conventional schemes. Further, the average number of
active SUs in each sensing time slot was 10.401, which
represents that all the SUs did not update their local sensing.
In other words, in the proposed approach, the optimal users
from correlation viewpoint are selected to update their own
local sensing. It will decrease the flow of data in the network
in updating local sensing results. Hence, the complexity of
the network will be less rather than the comparable DL-based
approach. Moreover, the presence of correlated users in
DL-based approach will increase the noise of measurements
and the number of neurons to be initialized, which will
decrease the accuracy of sensing according to the ROC.
Since the ROC of DL-based approach and DRL-based
proposed approach are almost identical, we have defined
active SUs which denotes the average number of cooperating
users in the algorithm. In DL-based approach, all the SUs
are cooperating to sense the unoccupied bands, while in the
proposed DRL-based approach the active cooperating SUs are
defined based on the correlation reward. This metric which
is depicted in Fig. 2, will show that in the network the
information exchange will be decreased while the ROC is
identical and the performance is not affected. This decrease
in the number of active users is the main advantage of the
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY WARANGAL. Downloaded on November 30,2022 at 14:20:54 UTC from IEEE Xplore. Restrictions apply.
IEEE COMMUNICATIONS LETTERS, VOL. 24, NO. 7, JULY 2020
Fig. 3. Sensing error vs. Tx power and number of samples.
proposed approach which will lower the information flow in
the network while the performance is not degraded.
Eventually, the sensing error is represented in Fig. 3, where
transmitting power and the number of samples are changed.
As depicted, the sensing error in both cases for the proposed
approach is more reliable than others. Specifically, two initial
results are the lowest sensing error and lowest fluctuations
in the sensing error by changing the number of samples and
transmission power.
V. C ONCLUDING R EMARKS
In this letter, the cooperative spectrum sensing is considered
in CRN by utilizing the machine learning techniques, specif-
ically DRL. Here, we have proposed a DRL based algorithm
to sense the spectrum holes, cooperatively. In this algorithm,
we have used RL to accurately choose the needed SUs to
cooperate and a DL in each SU, which is chosen to sense
the presence or absence of the PU locally. The proposed
approach is very suitable from the computational complexity
point since there is no need for the measurements of all
the SUs to be updated in each time slot. The reinforcement
part of the algorithm determines the required SU to share its
measurements and local sensing results.
R EFERENCES
[1] A. Hajihoseini Gazestani and S. Ali Ghorashi, “Distributed diffusion-
based spectrum sensing for cognitive radio sensor networks considering
link failure,” IEEE Sensors J., vol. 18, no. 20, pp. 8617–8625, Oct. 2018.
[2] A. W. Min and K. G. Shin, “An optimal sensing framework based on
spatial RSS-profile in cognitive radio networks,” in Proc. 6th Annu. IEEE
Commun. Soc. Conf. Sensor, Mesh Ad Hoc Commun. Netw., Rome, Italy,
Jun. 2009, pp. 1–9.
[3] K. M. Thilina, K. Won Choi, N. Saquib, and E. Hossain, “Machine
learning techniques for cooperative spectrum sensing in cognitive radio
networks,” IEEE J. Sel. Areas Commun., vol. 31, no. 11, pp. 2209–2221,
Nov. 2013.
[4] W. Lee, M. Kim, and D.-H. Cho, “Deep cooperative sensing: Cooperative
spectrum sensing based on convolutional neural networks,” IEEE Trans.
Veh. Technol., vol. 68, no. 3, pp. 3005–3009, Mar. 2019.
[5] W. Lee, M. Kim, and D.-H. Cho, “Deep power control: Transmit power
control scheme based on convolutional neural network,” IEEE Commun.
Lett., vol. 22, no. 6, pp. 1276–1279, Jun. 2018.
[6] M. Kim, N.-I. Kim, W. Lee, and D.-H. Cho, “Deep learning-aided
SCMA,” IEEE Commun. Lett., vol. 22, no. 4, pp. 720–723, Apr. 2018.
[7] X.-Y. Zhang, K. Zhang, X. Yun, S. Wang, X. Bao, and Q. Yuan,
“Location-based correlation estimation in social network via collaborative
learning,” in Proc. IEEE Conf. Comput. Commun. Workshops (INFOCOM
WKSHPS), San Francisco, CA, USA, Apr. 2016, pp. 1073–1074.
[8] Channel Models for IEEE 802.20 MBWA System Simulations,
document IEEE C802.20-03/70, Jan. 2007.
[9] A. Algans, K. I. Pedersen, and P. E. Mogensen, “Experimental analysis
of the joint statistical properties of azimuth spread, delay spread, and
shadow fading,” IEEE J. Sel. Areas Commun., vol. 20, no. 3, pp. 523–531,
Apr. 2002.