Future Generation Computer Systems ( ) –

Contents lists available at ScienceDirect

Future Generation Computer Systems

journal homepage: www.elsevier.com/locate/fgcs

Hybrid Directional CR-MAC based on Q-Learning with Directional

Power Control
Anil Carie a , Mingchu Li a , Chang Liu b , Prakasha Reddy c , Waseef Jamal d, *
a
School of Software, Dalian University of Technology, Dalian, China
b
School of Information and Communication Engineering, Dalian University of Technology, Dalian, China
c
Department of Informatics, College of Engineering and Technology, Wollega University T.R, Ethiopia
d
Center of Business Administration, Institute of Management Sciences, Hayatabad, Peshawar, Pakistan

highlights

• A cognitive radio concept is used to utilized channels in a more accurate way .

• Directional Hybrid control channel with GPS is used to exchange cognitive control information.
• Experimental results are carried out in context of proposed method in comparison with other existing schemes.

article info a b s t r a c t
Article history: In this paper, we investigate the Hybrid Directional CR-MAC based on Q-Learning with Directional
Received 8 August 2017 Power Control in cognitive radio (CR) systems. In CR systems, nodes can switch to heterogeneous non-
Received in revised form 20 October 2017 overlapping channels opportunistically which offer higher achievable throughput. However, the random
Accepted 7 November 2017
channel selection policy in existing CR-MAC protocol has problems like delay, packet collisions, and
Available online xxxx
quality of service. The proposed channel selection scheme which is quite different from the traditional
Keywords: scheme is adopted by nodes to achieve context awareness and intelligence for adaptive channel selection.
Software defined cognitive radio network The nodes select a channel based on the results learned by interactions with the other nodes and channels.
MAC protocol The directional transmission power control scheme allows the nodes to reuse the channels subject
Directional antennas to interference constraints. The simulation results show that nodes using the proposed algorithm can
Software defined radio select channels adaptively and optimal transmission power which helps to achieve high throughput and
Q-learning minimized power consumption.
Power control © 2017 Elsevier B.V. All rights reserved.

1. Introduction CR networks to answer scarcity problem. CR nodes can use Time

Fixed spectrum policies have led to underutilization of spec-
trum resources, one of the answers to spectrum scarcity problem
in a wireless network is Cognitive radio. Cognitive radio technol-
ogy enables nodes to dynamically access unused or underutilized
licensed spectrum opportunistically [1]. Therefore, spectrum uti-
lization significantly increases with spectrum aware CR network.
SU’s exploit space, frequency and time domains of radio spectrum
using models like opportunistic access [2–4] and concurrent ac-
cess [5–12]. CR devices can maximize throughput [2–4], minimize
energy consumption [10,11] and reduce interference [5–8] by ad-
justing their radio constraints. Spectrum sharing is important in
* Corresponding author.
E-mail addresses: carieanil@gmail.com (A. Carie), mingchul@dlut.edu.cn
(M. Li), cliu.wcom@hotmail.com (C. Liu), reddysinfo@gmail.com (P. Reddy),
waseef.jamal@imsciences.edu.pk (W. Jamal).
Division Multiple Access, Frequency Division Multiple Access or
Code Division multiple access or their combinations. But, there
exists fundamental question of competent channel selection and
optimal transmission power in the dynamic time-varying envi-
ronment, i.e. channel characteristics, the mobility of the wireless
sources, the wireless nodes unpredictability in joining the network
or leaving the network etc. The MAC protocols in the literature [13]
can work as guidelines for Channel switching according to envi-
ronmental changes in cognitive radio networks. However, the cost
of frequency changes in current manufactured wireless devices is
not considered by most of them. On an average channel, switching
can cause packet loss ratio up to 3% exclusively, which can be
even worse in cognitive radio networks due to dynamic nature
of primary users. Thus optimal channel selection strategy which
fluctuates less frequently is favorable. In [14] power control poli-
cies are devised for cellular networks with QoS as primary criteria,
where transmitters increase power to deal with interference and

https://doi.org/10.1016/j.future.2017.11.014
0167-739X/© 2017 Elsevier B.V. All rights reserved.

Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
2 A. Carie et al. / Future Generation Computer Systems (
channel impairments. However, it is not suitable for primary user’s
QoS if secondary users transmit with arbitrarily high power. Thus
it is natural that power control should rely on the interference
levels. In this paper, we propose ‘‘Hybrid Directional CR-MAC based
on Q-Learning with Directional Power Control’’ where learning
algorithm is used for channel selection and a new Directional
transmission power control scheme (DTPC) for enhancing the
throughput and energy efficiency of the directional hybrid CR MAC
protocol. The main contributions of our work are twofold. First,
using channel selection algorithm we try to select the best channel
based on SU’s observation of PU’s traffic, channel characteristics
like throughput achieved, packets lost. Second we investigate the
problem of channel reuse in directional communication, where
CR user has the power control capability. That is, CR user can
transmit at any transmit power in the allowable power range that
to achieve maximum concurrent transmissions. The organization
of the rest of the paper is as follows. In Section 2, we present
related work. In Section 3 we present system model. Overview of
the Q-learning algorithm is presented in Section 4. The proposed
Hybrid Directional CR-MAC based on Q-Learning with Directional
Power Control Scheme is presented in Section 5. Simulation results
for different network topologies are shown in the Section 6 to
establishes the substantial throughput and energy gains that can
be attained under the investigated scheme. Finally, we present our
conclusions and future work.
2. Related work
Game theory based CR
In CR networks Game theory has recently been the most pop-
ular method for attaining context-awareness and intelligence. In
which, SU’s interact to maximize their individual objective such
as delay, throughput etc. however, there are several limitations
in game theory which are addressed using RL approach. Firstly,
GT based CR requires a complete set of information to compute
the Nash equilibrium; hence it is more suitable for centralized CR
networks [15,16].
Secondly, GT assumes a single type of objective function
throughout the CR network, and hence a homogeneous learning
mechanism in all the SUs. Thirdly, SU’s might converge to sub
optimal action due to miss-coordination even when optimal ac-
tion exists. Although the GT has been successfully applied in CR
networks [17–26], the RL approach is a good alternative which
addresses the issues above associated with GT. For instance, the
RL supports heterogeneous learning mechanism in each agent
because each agent can represent distinctive performance metrics
as local rewards.
Omni directional Power control
SU’s vary their transmit power depending on interference
at primary receiver and maximum secondary transmit power
constraint [27]. Concurrent transmission region is maximized
in [6] using optimal power control. The number of concurrent
transmissions are maximized in [7] using dynamic spectrum shar-
ing. With objectives of maximizing sum-rate, achieving rate fair-
ness, minimizing power consumption using power control are
studied in [28–32]. The necessary and sufficient condition for the
feasible region using Power controlled MAC consisting of only
two transmission links is derived in [33]. Channel hopping se-
quence is used to allocate the control channel to one-hop neighbor
nodes [25]. The basic drawback of sequential CCC based CR-MAC is
longer channel rendezvous delays [26–31]. Channel rendezvous is
more challenging for increased availability of PU channels.
Directional power control
For reusing spectrum in the macro cell, underlying microcell
uses Antenna beamforming and power allocation schemes to max-
imize multiuser sum rate [34]. Capacity and power consumption
Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
) –
of wireless network using directional antennas is studied in [35].
In [36], the achievable throughput of mobile ad hoc network with
the directional antenna is addressed. Directional Medium Access
Control is studied in [37], it suffers from deafness and hidden
terminal problems. First Power control with a directional antenna
over packet radio network was considered in [38]. Power control
built on D-MAC with Directional RTS, Omni Directional CTS and
optimal power for the data packet is studied in [39], increased
network capacity and reduced power consumption. An optimiza-
tion problem for selecting the range of channels for transmission
with the control channel and aggregated data channel was em-
ployed in statistical channel allocation MAC (SCA-MAC) [40] which
outperforms random scheme. DSA-MAC [41], DCRMAC [42], HC-
MAC [43], DDMAC [44], SMA [45] are the protocols which are
similar to IEEE802.11 DCF standard for reserving a channel.
3. System model
In this study, users with CR capabilities, referred to as sec-
ondary users, can communicate with other CR nodes utilizing the
primary networks available spectrum spatially and/or temporally.
When the secondary network doesn’t have enough resources CR
nodes form an ad-hoc network without a central controller or
dedicated control channels. Due to highly dynamic and hetero-
geneous networking environment, a dedicated control channel
is not pre-defined for exchanging control messages. We assume
that the nodes are close enough as to consider an interference-
limited spectrum sharing scenario in which a CRAHN operates.
The system model is composed of M licensed channels which
are accessed opportunistically by K CR nodes (acts as both trans-
mitter and receiver), D primary users. The primary transmitter,
primary receiver, and the mobile CR devices are distributed in
randomly within the coverage area. Similar to [46–48], the two-
state continuous-time Markov process is used to model the traffic
of each channel: Channel occupied by the PU (busy state) and the
channel that is not occupied by the PU(idle state). These two states
are referred as ON and OFF respectively. Each SU transmitter and its
corresponding SU receiver, but also on the time-varying activities
of the PUs. We consider the situation that several SUs may compete
for the same channel, and one SU may have more than one channel
for selection
Antenna model
To predict the received power, as in [49,50], we consider a gen-
eral power propagation model Pr = Pt CG rt Gtr
dtrα
where Pt is the trans-
mit power, dtr is the distance between transmitter and receiver,
α is the path-loss exponent, Gtr and Grt are the directive gains of
the transmitting and receiving antennas toward the direction of
the receiving and transmitting antennas, respectively, while C is a
constant determined by other factors as antenna heights and wave
length.
Time slot structure
The system model has slotted transmission structure as shown
in Fig. 3 and described as follows Each secondary user executes
following stages synchronously during each time slot.
- Channel Sensing: SU’s sense the PU channel’s to detect the
activity of PU’s.
- PCL-EXCHANGE: After Sensing SU’s broadcasts their Primary
user free Channel List (PCL) to its neighbors. After receiving
PCL information from neighbors, SU’s update PCL table.
- CHANNEL RESERVATION: it is divided into N slots and every
slot is divided in to two sub slots sub slot (S2) for a node to
send RTS directionally and sub slot (S2) for the destination
node to reply with CTS or DNAV.
- DATA TRANFER: SU’s which successfully reserved channel in
CHANNEL RERVATION PHASE start data transmission
 A. Carie et al. / Future Generation Computer Systems (
4. Overview of Q-learning algorithm
We use Q-Learning [12], which is a recent form of reinforcement
learning algorithm that does not need a model of its environment
and works by estimating the values of state–action pairs. The
estimate of future reward value in Q- learning algorithm is given
by Q(S,A) when an agent takes a particular action A when in a
particular state S. Learning intervals are denoted by t∈T = {1,2,. . . },
a constant interval by tD , actions by a∈A, rewards by rt+1(at). Every
agent records then learnt values from the environment in Q-table
for all its possible actions with | A| entries. The local reward for
an action is reflected through in its Q -value; hence agents change
their actions when there is a change in Q -value. At each interval
t, agent i chooses an action at, and receives local reward rt+1 (at )
at time t+1. The agent i updates the Q -value of at at time t+1 as
follows:
Qti+1 (att ) ← (1 − α )Qti (ait ) + α rti+1 (ait )
where 0≤α≤1 is learning rate. Value of α decides the dependence
on the reward, a higher value of α gives more importance to local
reward than past knowledge.
Agents search for action that maximizes value function Vπ as
shown below:
V π = max(Qti (a)).
a∈A
By exploring the environment, the agents build a table of Q -
values for each environment state and each possible action. Ex-
ploitation chooses the best known action, or the greedy action, at
all times for performance enhancement. Exploration chooses the
other non-optimal actions once in a while to improve the estimates
of all the Q -values in order to realize better actions. The learning
rate and the discount factor are important parameters of the Q-
learning algorithm. The learning rate parameter limits how quickly
learning can occur. The discount factor controls the value placed on
future rewards.
5. Hybrid directional CR-MAC based on Q-learning with direc-
tional power control
5.1. Channel selection using Q-learning
We present our directional antenna based hybrid CCC based
CR-MAC with dynamic channel selection implementation in this
section. Each SU node is equipped with one transceiver which is
used for both control and data. 902MHz is used by nodes to ex-
change their free channel list which is used to find common control
channel (CCC) among nodes. Nodes decide on PU free (available)
data channel in the licensed band over CCC. Two-way handshaking
is performed by nodes to transmit control and data information.
An illustration of the cognitive MAC protocol is shown in Fig. 4.
Channel switching decision is made at the beginning of each time
slot which depends on the channel state information.
In Directional Hybrid (DH) CR-MAC the DCS applies QL to select
a channel. Each agent divides time in to fixed intervals of ‘t’ and
keeps track of a number of packets transmitted successfully ‘ND ’.
At the beginning of each‘t’ every node updates Q -value using
(1) and chooses the channel with highest Q -value and broadcast
to its neighbors along with PCL. Nodes after receiving broadcast
information update PCL table and calculate population state of
each channel, i.e. number of users selecting a given channel. Nodes
select the channel with least population state for the transmission.
Every node maintains Q-table that consists of Q -values which are
in the range of 0 to 1. We use dynamic Q-table that the size of
Q-table of the node is determined by the number of available
channels. The Q-table and the learning task are distributed among
Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
) – 3
the different nodes (states). Therefore, when choosing the next
channel, we let agent act greedily, taking, in each situation, the
action with highest Q -value. If the node can transmit data suc-
cessfully using the channel, then reward R will be increased by the
number of packets transmitted otherwise R will be 0. The discount
factor is an important parameter of the Q-Learning algorithm. We
use variable discount factor which is determined by (PU activity
rate, number of SU competing for the channel, bandwidth).
CHANNEL SELECTION USING DH-CRMAC
Set α ϵ [0, 1] //learning rate
For initial time slot
Select random channel
Broadcast PCL with chosen channel ID’s at the top of the list
to other users.
Receive the information of other user’s channel selection.
Calculate population state of each channel (Count number of
user selecting a given channel).
Choose the channel with least count.
(1) Sense and contend for the chosen channel and
Transmit data packets if successfully grabbing the channel.
If (receive ACK for the DATA packet sent)
Then ND = ND+1
End if
End of initial time slot.
For remaining time slot’s
rt+1 (at ) = (ND/TD) + population state
(2) Q t+1 (at ) = (1 − α ) Qt (at ) + α rt+1 at ).
R = uniform (0, 1) {generate random number}
If R<=ε then
a temp = uniform (1, k)
Else
a temp = Argmax a ∈ A (Q (a))
If | Q t+1 (a temp)− Qt+1 (at )| <= β then
a t+1 = at
Else
a t+1 = atemp
End if
End if
Return a t+1
End if
Broadcast PCL with chosen channel ID’s at the top of the list other
users.
Receive the information of other user’s channel selection.
Calculate population state of each channel (Count number of the user
selecting a given channel).
Choose the channel with least count.
Sense and contend for the chosen channel and
Transmit data packets if successfully grabbing the channel.
End of time slot
5.2. Optimal directional power control
In this section, the proposed optimal directional power control
algorithm for channel spatial reuse is presented. We first study
the feasibility of the channel reuse with proposed optimal power
control algorithm. Fig. 2 illustrates a classical spectrum access or
spectrum sharing scenario with D randomly distributed primary
users (PU in Fig. 2) and K secondary users (SU in Fig. 1). In this
scenario, we assume that each of the PUs is equipped with Omni-
antenna, while each of SUs is equipped with multi-antenna, which
is available for beamforming technology. In this case, the small
cell consists of PU broadcasting channels and SU beamforming
channels. In addition, considering the mobility of both the primary
users and secondary users, we assume that both PUs and SUs follow
the homogeneous Poisson point process (HPPP). Let {N(A)} denotes
the number of users in the area ‘‘A’’, such as the cell in Fig. 2. If {N
(A)} follows an HPPP with the intensity of λ > 0, that is, N(A) ∼
Poisson (λ| A|), then the probability of N(A)= k can be expressed
as:
e−λ|A| (λ|A|)k
P(N(A) = k) = . (3)
k!
4 A. Carie et al. / Future Generation Computer Systems ( ) –

Fig. 3. Antenna, channel and node reservation tables in directional CR-MAC.

Fig. 1. Cognitive radio network with learning and power control.

Fig. 4. Node communication using hybrid control channel using learning and power
control.

Fig. 2. Illustration of distributed spectrum access with spatial reuse.

Based on the discussion above, we can then analyze the access

scenario. As for the PUs in the cell, although the directional beams
are adopted among the SUs communications, the PU receivers may
still suffer the interferences from the SUs’ inaccuracy beamform-
ing as the broadcasting channels can receive signals from all the
directions in space. As for the SUs in the cell, by adjusting the
beamforming direction, the SU system can nullify the interference
to PU systems, but the SU system needs to know the interference
channel information to control the interference by beamforming.
We assume that cooperation exists between the PU system and
SU system, and cooperation could be achieved by using a direct
feedback wireless channel link from the PU users to the SUs. We
skip the details of the feedback protocols between the PU receiver
and SU transmitter, as this is out of focus of this paper. According
to the discussion above, we can then make a mathematical formu-
lation for the channel and signal model, which is generalized as
the application model in Fig. 5. Note that in this model, a single
SU receiver coexists with D PU receivers, while sharing the same
radio resources. The SU transmitter has M antennas, while both
the SU receivers and PU receivers are equipped with a single
antenna. Thus, the signal received at the SU receiver,y ∈ C and
the interference to the ith PU receiver from the SU transmitter, Ii
are, respectively, given by
y = hH x + n
2 2
Ii = |gH
i x|
Fig. 5. Interference of node SU to neighboring PU’s.
SU receiver and transmitter, and gi ∈ C M ×1 is the interference
channel vector from the SU transmitter to the ith PU receiver. The
transmitted signal is given by x = v · s, meaning that a single
dimensional symbol, s ∈ C , is weighted by a complex beamforming
vector v ∈ C M ×1 . The transmit signal must satisfy the transmit
power constraint of PM , i.e., E[∥x∥2 ] ≤ PM . Then, by restricting
our attention to the normalized beamforming vector v, where
∥v∥2 = 1, we can limit the signal power P by E[| s| 2| ]= P ≤P M .
Our problem is to determine the optimal beamforming vector and
transmit power such that the transmission gain in the SU link is
maximized while the interference to the PU receiver is constrained
to a predefined threshold. Mathematically, this problem can be
described as
2
(4) max γs = P |hH v|
v,P (6)
(5) s.t .Ii = |gH
i v| ≤ Ith , ∥v∥ = 1, and P ≤ PM .
2

where n is a zero-mean independent complex Gaussian noise with For, i = 1, 2, . . . , m where γs is the channel gain of the SU link.
unit variance; h ∈ C M ×1 is the channel vector between the Therefore, by solving the above optimization, we can finally obtain

Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
 A. Carie et al. / Future Generation Computer Systems ( ) – 5

Table 1
Simulation parameters.
Parameter name Description
Topology 1000 *1000 Flatgrid
No. of CR nodes 100
No. of PU channels 10(8 MHz) channels
No. of PU transmitters 10
Unlicensed channel ISM-902 MHz
PU active probability 10, 15 and 20 msec
Mobility model Random waypoint
Input CR transmit power 10 mW
Receiver threshold −95 dbm
Carrier Sense threshold −115 dbm
CR Tx range 200 m (Licensed channel)
PU Tx range 500 m (Licensed channel)
Data rate 2 Mbps
Antenna type(Channel reservation) Directional (4-element)
Antenna type(Data transmission) Directional(4-element)
Beamwidth 900
Interface Queue length 50
Simulation time (s) 100 Sec
Traffic type CBR/UDP
Packet size (bytes) 512 & 1024 bytes

the optimal beam shape to achieve the directional transmission

among SUs with tolerance interference to PUs.

6. Simulation results

Cognitive radio network simulator (NS-2.31) is used to imple-

ment the proposed reinforcement learning based CR-MAC proto-
col. Our primary goal is to show learning based channel selection,
and optimal power control would improve throughput, reduce
channel switching and improve overall performance. Also, the pro-
posed Cognitive MAC protocol is compared in terms of aggregate
network throughput, average packet delay, per-hop delay with
respect to data rate, PU spectrum occupancy rate and PU channel
availability. Furthermore, the proposed Cognitive MAC protocol
with respect to reinforcement learning is compared with In-band
Cognitive MAC, out-of-band Cognitive MAC and hybrid CCC MAC
protocols. The simulation parameters that are used in this paper is
shown in Table 1. A flat-grid 1000*1000 topology is used to run the
network simulations. It is noted that 100 CR nodes are considered
for this simulation with 10 PU channels and 10 PU base stations.
Fig. 6(a) describes energy consumption for application data with
respect to number of CR nodes. Whenever, the number of cognitive
nodes within the network increases then there is an increase in
collision probability due to more traffic contention.
Fig. 6(a) describes energy consumption for application data
with respect to number of CR nodes. Whenever the number of
cognitive nodes within the network increases then there is an
increase in collision probability due to more traffic contention. This
is more common in opportunistic cognitive radio access network
due to contention with CR nodes and evacuation due to PU active
in current CR communication channel. From Fig. 6(a), it is clear
that the proposed Cognitive MAC protocol will have less energy
consumption with respect to in-band and out-of-band CCC based
CR networks.
In Fig. 6(b) we have shown the relation of aggregate throughput
to number of CR nodes and observed that CR nodes with learning
and without learning have similar throughput initially. CR nodes
with learning at first do not have knowledge of PU activity. How-
ever, over a period as CR nodes determine the usage pattern of
primary users, i.e., PU’s they select PU free channels that would Fig. 6. (a) Per-hop control energy consumption with respect to No. of CR nodes. (b)
yield higher throughputs. CR nodes would stop choosing channels Per-hop data energy consumption with respect to No. of CR nodes. (c). Aggregate
Network Throughput with respect to No. of available PU channels. (d) Aggregate
which are frequently used by PU’s. So from Fig. 6(b), it is evident
Network Throughput with respect to Spectrum Occupancy rate.
that learning helps to achieve higher throughputs.

Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
6 A. Carie et al. / Future Generation Computer Systems ( ) –

Fig. 6(c) shows the curves between aggregate network through-

put and number of available PU channels. For this, we have con-
sidered negligible sensing time. CR nodes learn about the channel
usage pattern of PU’s and nearby CR nodes and select channels
which give the maximum period to transmit without interrup-
tions. Learning helps to avoid competition with PU and other CR
nodes which contribute to promote utilization of all the available
channels. Finally, Fig. 6(d) shows an increase in throughput as the
number available PU free channels increases
Fig. 7(a) shows aggregate throughput as function of MAC data
rate. We have taken number of PU channels as 5 and data size as
1024 bytes. With the increase in data rate CR nodes compete to
access the channel with the higher data rate, but CR nodes also
understand through learning mechanism that other nodes also
want to access the high data rate channel. Thus they tend to choose
a channel which satisfies their requirement hence avoiding colli-
sion with other CR nodes. In Fig. 7(c) the number of PU channels
was increased to 10, which will increase transmission probability.
Now CR nodes will shift to channel which satisfies its throughput
requirements.
Average packet delay of the proposed scheme is presented and
verified in Fig. 7(b). Herein we have analyzed performance by
setting a number of PU channels as 5. Packet delay increases as
the collisions increase. By comparison, proposed CR-MAC utilizing
learning scheme and also optimal power decreases the number
of collisions, which in turn decrease the packet delay. We have
compared the performance of packet delay with state of the art CR-
MAC protocols. In Fig. 7(d) the number of PU channels are increased
to 10 which shows a direct impact on packet delay. CR nodes
which are waiting in the queue or in back off phase can switch into
new transmission opportunities. Unlike CR nodes without learning,
CR nodes with learning are more adaptive to dynamic channel
opportunities.

7. Conclusion and future work

Enhancing the network throughput with minimal node energy

consumption is significant in cognitive radio ad-hoc networks.
This paper proposes A Hybrid Directional CR-MAC based on
Learning with Power Control for ad-hoc cognitive radio networks
that uses Q-learning scheme for channel selection based on the
observation of PU’s traffic and other SU’s channel selection. The
protocol learns the neighbor nodes channel selection and adapts to
channel with less chance of collision. In addition, node calculates
optimal power for transmission, which increases spatial reuse
thus improving channel utilization and systems throughput. The
proposed schemes is shown to have better performance compared
with the non-learning CRMAC protocols. In the future work real
time application based channel scheduling will be considered in
accordance with the proposed scheme.

Acknowledgment

This work was supported by Nature Science Foundation of china

under grant number : 61572095. Also The authors would like to
be grateful the anonymous reviewers for their valuable comments
and suggestions to enhance the quality of the paper.

References

Fig. 7. (a) Aggregate Throughput with respect to data rate(PU channels :5) (b)
Average packet delay with respect to data rate(PU channels : 5) (c). Aggregate
Network Throughput with respect to data rate (PU channels : 10) (d) Average packet
delay with respect to MAC data rate (PU channels: 10).
[1] J. Mitola, Cognitive radio: An integrated agent architecture for software de-
fined radio (Ph.D. dissertation), KTH Royal Institute of Technology, 2000.
[2] S. Srinivasa, S.A. Jafar, Soft sensing and optimal power control for cognitive
radio, IEEE Trans. Wireless Commun. 9 (12) (2010) 3638–3649.
[3] V. Asghari, S. Aissa, Adaptive rate and power transmission in spectrum-sharing
systems, IEEE Trans. Wireless Commun. 9 (10) (2010) 3272–3280.

Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
 A. Carie et al. / Future Generation Computer Systems (
[4] E.C.Y. Peh, Y.-C. Liang, Y. Zeng, Sensing and power control in cognitive radio
with location information, in ICCS, 2012, pp. 255–259.
[5] L.-C. Wang, A. Chen, Effects of location awareness on concurrent transmissions
for cognitive ad hoc networks overlaying infrastructure-based systems, IEEE
Trans. Mobile Comput. 8 (5) (2009) 577–589.
[6] Y. Song, J. Xie, Optimal power control for concurrent transmissions of location-
aware mobile cognitive radio ad hoc networks, in GLOBECOM, no. July 2009,
pp. 1–6.
[7] M.R. Hassan, G. Karmakar, J. Kamruzzaman, Maximizing the concurrent trans-
missions in cognitive radio ad hoc networks, in IWCMC, no. July 2011, pp. 466–
471.
[8] S.M. S’anchez, R.D. Souza, E.M.G. Fernandez, V.A. Reguera, W. Godoy, Effect of
location accuracy and shadowing on the probability of non-interfering con-
current transmissions in cognitive ad hoc networks, Radioengineering 22 (4)
(2013) 1138–1149.
[9] S.M. S’anchez, R.D. Souza, E.M.G. Fernandez, V.A. Reguera, Impact of rate
control on the performance of a cognitive radio adhoc network, IEEE Commun.
Lett. 16 (9) (2012) 1424–1427.
[10] S.M. Sanchez, R.D. Souza, E.M.G. Fernandez, V.A. Reguera, Rate and energy
efficient power control in a cognitive radio ad hoc network, IEEE Signal Process.
Lett. 20 (2013) 451–454.
[11] S. Buzzi, D. Saturnino, A game-theoretic approach to energyefficient power
control and receiver design in cognitive CDMA wireless networks, IEEE J. Sel.
Topics Signal Process. 5 (1) (2011) 137–150.
[12] M. Ku, L. Wang, Y.T. Su, Toward optimal multiuser antenna beamforming for
hierarchical cognitive radio systems, IEEE Trans. Commun. 60 (10) (2012)
2872–2885.
[13] Proceedings of the first IEEE symposium on New Frontiers in Dynamic Spec-
trum Access Networks, November 2005.
[14] M. Chiang, P. Hande, T. Lan, C.W. Tan, Power control in wireless cellular
networks, Found. Trends Netw. 2 (4) (2008) 381–533.
[15] Z. Ji, K.J.R. Liu, Dynamic spectrum sharing: A game theoretical overview, IEEE
Comm. Mg. 45 (5) (2007) 88–94.
[16] D. Niyato, E. Hossain, Competitive spectrum sharing in cognitive radio
networks: A dynamic game approach, IEEE T. Wls. Comm. 7 (7) (2008)
2651–2660.
[17] N. Nie, C. Comaniciu, Adaptive channel allocation spectrum etiquette for
cognitive radio networks, Symp. on New Frntr. in Dynmc. Spctrm. Acs. Nwk.
(DySPAN), IEEE, Baltimore, MD, 2005, pp. 269–278.
[18] M.R. Musku, A.T. Chronopoulos, S. Penmatsa, D.C. Popescu, A game theoretic
approach for medium access of open spectrum in cognitive radios, 2nd Intl.
Conf. on Cgntve. Rd. Orntd. Wls. Nwk. and Comm. (CROWNCOM), IEEE, Or-
lando, FL, July 2007, pp. 336–341.
[19] Z. Han, C. Pandana, K.J.R. Liu, Distributive opportunistic spectrum access for
cognitive radio using correlated equilibrium and no-regret learning, Wls.
Comm. Nwk. Conf. (WCNC), IEEE, Hong Kong, March 2007, pp. 11–15.
[20] Z. Ji, K.J.R. Liu, Dynamic spectrum sharing: A game theoretical overview, IEEE
Comm. Mg. 45 (5) (2007) 88–94.
[21] S. Subramani, T. Basar, S. Armour, D. Kaleshi, Z. Fan, Noncooperative equilib-
rium solutions for spectrum access in distributed cognitive radio networks,
Symp. On New Frntr. in Dynmc. Spctrm. Acs. Nwk. (DySPAN), IEEE, Chicago, IL,
October 2008.
[22] H.N. Pham, J. Xiang, Y. Zhang, T. Skeie, QoS-aware channel selection in cog-
nitive radio networks: A game-theoretic approach, Glbl. Telecomm. Conf.
(GLOBECOM), IEEE, New Orleans, LA, December 2008.
[23] H. Qin, H. Wang, H. Zhou, A selfish game-theoretic approach for cognitive radio
networks with dynamic spectrum sharing, Intl. Conf. on Comp. Sc. Sftwr. Eng.
(CSSE), China, December 2008, pp. 1105–1109.
[24] X. Gong, W. Yuan, W. Liu, W. Cheng, S. Wang, A cooperative relay scheme for
secondary communication in cognitive radio networks, Glbl. Telecomm. Conf.
(GLOBECOM), IEEE, New Orleans, LA, December 2008.
[25] V. Maskery, V. Krishnamurthy, Q. Zhao, Decentralized dynamic spectrum
access for cognitive radios: Cooperative design of a noncooperative game, IEEE
T. Comm. 57 (2) (2009) 459–469.
[26] H.-P. Shiang, M.V.D. Schaar, Queuing-based dynamic channel selection for
heterogeneous multimedia applications over cognitive radio networks, IEEE
T. Mm 10 (5) (2008) 896–909.
[27] S. Srinivasa, S.A. Jafar, Soft sensing and optimal power control for cognitive
radio, IEEE Trans. Wireless Commun. 9 (12) (2010) 3638–3649.
[28] A. Hoang, Y. Liang, M. Islam, Power control and channel allocation in cognitive
radio networks with primary users’ cooperation, IEEE Trans. Mob. Comput.
9 (3) (2010) 348–360.
Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
) – 7
[29] B. Maham, P. Popovski, X. Zhou, A. Hjorungnes, Cognitive multiple access net-
works with outage margin in the primary system, IEEE Trans. Wirel. Commun.
10 (10) (2011) 3343–3353.
[30] E. Dall’Anese, S.-J. Kim, G.B. Giannakis, S. Pupolin, Power control for cognitive
radio networks under channel uncertainty, IEEE Trans. Wirel. Commun. 10 (10)
(2011) 3541–3551.
[31] X. Gong, S. Vorobyov, C. Tellambura, Optimal bandwidth and power allocation
for sum ergodic capacity under fading channels in cognitive radio networks,
IEEE Trans. Signal Process. 59 (4) (2011) 1814–1826.
[32] C.C. Zarakovitis, N. Qiang, D.E. Skordoulis, M.G. Hadjinicolaou, Power-efficient
cross-layer design for OFDMA systems with heterogeneous QoS, imperfect CSI,
and outage considerations, IEEE Trans. Veh. Technol. 61 (2) (2012) 781–798.
[33] A. Behzad, Z. Rubin, Multiple access protocol for power-controlled wireless
access nets, IEEE Trans. Mob. Comput. 3 (4) (2004) 307–316.
[34] M. Ku, L. Wang, Y.T. Su, Toward optimal multiuser antenna beamforming for
hierarchical cognitive radio systems, IEEE Trans. Commun. 60 (10) (2012)
2872–2885.
[35] P. Li, C. Zhang, Y. Fang, The capacity of wireless ad hoc networks using
directional antennas, IEEE Trans. Mobile Comput. 10 (10) (2011) 1374–1387.
[36] Y. Chen, J. Liu, X. Jiang, O. Takahashi, Throughput analysis in mobile ad hoc
networks with directional antennas, Ad Hoc Networks 11 (3) (2013) 1122–
1135.
[37] R. Choudhury, N.H. Vaidya, Impact of Directional Antennas on ad hoc Routing.
In Conference on Personal and Wireless Communication, September 2003.
[38] J. Zander, Slotted Aloha multihop packet radio networks with directional
antennas, Electron. Lett. 26 (25) (1990).
[39] Y.B. Ko, V. Shankarkumar, N.H. Shankarkumar, Medium access control proto-
cols using directional antennas in ad hoc networks. In Annual Joint Conference
of the IEEE Computer and Communications Societies, March 2000.
[40] A. Chia-Chun Hsu, David S.L. Wei, C.-C. Jay Kuo, A cognitive radio MAC protocol
using statistical channel allocation for wireless ad hoc networks, In Proc. IEEE
WCNC, March 2007, pp. 105–110.
[41] S.L. Wu, C.Y. Lin, Y.C. Tseng, J.P. Sheu, A new multi-channel MAC protocol with
on-demand channel assignment for multi-hop mobile ad hoc networks, IEEE
DySPAN, Maryland, USA, 2005, pp. 203–213.
[42] S.J. Yoo, H. Nan, T.I. Hyon, DCR-MAC: Distributed cognitive radio MAC protocol
for wireless ad hoc networks, WirelCommun Mobile Comput. 9 (5) (2009) 631–
653.
[43] J. Jia, Q. Zhang, X. Shen, HC-MAC: A hardware-constrained cognitive MAC for
efficient spectrum management, IEEE J. Sel. Areas Commun. 26 (1) (2008) 106–
117.
[44] H.A.B. Salameh, M.M. Krunz, O. Younis, Cooperative adaptive spectrum
sharingin cognitive radio networks, IEEE/ACM Trans. Netw. 18 (4) (2010)
1181–1194.
[45] X. Wang, A. Wong, P.H. Ho, Stochastic medium access for cognitive radio ad
hoc networks, IEEE J. Sel. Areas Commun. 29 (4) (2011) 770–783.
[46] Y. Yilmaz, Z. Guo, X. Wang, Sequential joint spectrum sensing and channel
estimation for dynamic spectrum access, IEEE J. Sel. Areas Commun. 32 (11)
(2014) 2000–2012.
[47] N. Khambekar, C.M. Spooner, V. Chaudhary, On improving serviceability with
quantified dynamic spectrum access, in Proceedings of the IEEE International
Symposium on Dynamic Spectrum Access Networks (DySPAN ’14), pp. 553–
564, McLean, Va, USA, April 2014.
[48] T.M.C. Chu, H. Phan, H.J. Zepernick, Hybrid interweave underlay spectrum
access for cognitive cooperative radio networks, IEEE Trans. Commun. 62 (7)
(2014) 2183–2197.
[49] P. Li, C. Zhang, Y. Fang, The capacity of wireless ad hoc networks using
directional antennas, IEEE Trans. Mobile Comput. 10 (10) (2011) 1374–1387.
[50] Y. Chen, J. Liu, X. Jiang, O. Takahashi, Throughput analysis in mobile ad hoc
networks with directional antennas, Ad Hoc Networks 11 (3) (2013) 1122–
1135.
Anil Carie received the B.Tech. Degree in Computer Sci-
ence and Engineering from Jawaharlal Nehru Technolog-
ical University, Hyderabad, India and M-Tech degree in
software Engineering from Karunya University, Coimbat-
ore, India. He worked as faculty in Vidya Barathi Insti-
tute of Technology, Warangal, India from Jun-2010 to
Jun-2014. He is currently a Ph.D. candidate in School of
Software Technology of Dalian University of Technology,
China. His research interests include Common control
channel design for MAC and routing protocols in Cognitive
radio ad-hoc networks.
8 A. Carie et al. / Future Generation Computer Systems (
Mingchu Li received the B.S. degree in mathematics,
Jiangxi Normal University and the M.S. degree in applied
science, University of Science and Technology Beijing in
1983 and 1989, respectively. He worked for University of
Science and Technology Beijing as an associate professor
from 1989 to 1994. He received his doctorate in Math-
ematics, University of Toronto in 1998. He worked for
School of Software of Tianjin University as a full professor
(from 2002 to 2004). and, from 2004 to now, for School of
Software Technology of Dalian University of Technology as
a full Professor and Vice Dean. His main research interests
include theoretical computer science and information security, and trust models
and cooperative game theory.
Chang Liu received the B.S. degree in electronic informa-
tion engineering from Dalian Maritime University, Dalian,
China, in 2012, and the Ph.D. degree in signal and informa-
tion processing, Dalian University of Technology, China.
From 2015 to 2016, he was a visiting scholar in depart-
ment of electrical engineering and computer science at
University of Tennessee, Knoxville, USA. He is currently a
postdoctoral research fellow with the National Key Lab-
oratory of Science and Technology on Communications,
University of Electronic Science and Technology of China,
Chengdu, China, 611731. His research interests include
Spectrum Sensing in Cognitive Radio, Statistical Signal Processing, Random Matrix
Theory and Array Signal Processing.
Please cite this article in press as: A. Carie, et al., Hybrid Directional CR-MAC based on Q-Learning with Directional Power Control, Future Generation Computer Systems
(2017), https://doi.org/10.1016/j.future.2017.11.014.
) –
CH.R. Prakasha Reddy is currently working as a Lecturer
at College of Engineering and Technology in Department of
Informatics, Wollega University T.R, Ethiopia. He received
his Bachelor degree in Information Technology (2007)
from Department of Information Technology, St.Theressa
Institute of Engineering and Technology, Jawaharlal Nehru
Technological University, Andhra Pradesh, India. He Joined
Karunya University Coimbatore, India for Masters in Net-
work and Internet Engineering (2010). He has worked as
Assistant Professor at Department of Information Tech-
nology, SRK Institute of Technology, Vijayawada, Andhra
Pradesh, (2010 – 2013) India. His main Research interests include Storage Area Net-
works, Network security, Energy Efficient Routing in wireless Networks, Artificial
Intelligence.
Waseef Jamal is an Assistant Professor at the Center of
Business Administration, Institute of Management Sci-
ences, Hayatabad, Peshawar, (Pakistan). His research in-
terest is in the area of organization behavior and Invest-
ment in Human capital. Dr. Waseef has no of national and
international journal publication and presented papers
at both local and international conferences. Additionally,
he is also a Trainer, Consultant and philanthropist and
member of environmental committee at Institute of Man-
agement Sciences.