IEEE ON IEEE RELIABILITY SOCIETY SECTION

Received January 2, 2021, accepted January 23, 2021, date of publication January 26, 2021, date of current version February 17, 2021.
Digital Object Identifier 10.1109/ACCESS.2021.3054756

The Performance Evaluation of Big Data-Driven

Modulation Classification in
Complex Environment
ZHUORAN CAI 1, JIDONG WANG 2, AND MINGHUAN MA 3
1 School of Opto-Electronic Information Science and Technology, Yantai University, Yantai 264005, China
2 Wuhan Maritime Communication Research Institute, Wuhan 430079, China
3 College of Information and Communication, Harbin Engineering University, Harbin 150001, China

Corresponding author: Minghuan Ma (mh_m@hrbeu.edu.cn)

ABSTRACT With the proliferation of frequency-using devices and the advent of the era of big data, spectrum
management and control are faced with challenges of effectiveness and accuracy. Modulation classification
technology is the foundation and key part of spectrum management and control. Therefore, the effectiveness
of modulation classification technology in big data scenario is very important. In this paper, we consider
not only the validity of the classification model under the background of big data, but also the dynamics
of noise in the complex electromagnetic environment. So we construct a big dataset containing different
signals under different MSNR, and use the big data to drive the deep learning model, and finally get the
classification result. The proposed method can realize modulation classification only by training one model,
which avoids the redundancy of model training in previous algorithms. The simulation results demonstrate
the effectiveness and reliability of the proposed method.

INDEX TERMS Big data, non-Gaussian noise, modulation classification, deep learning.

I. INTRODUCTION function values of different signals, so as to obtain a large

Automatic modulation classification (AMC) is the key part of
physical layer transmission technology, and the signal mod-
ulation types classified pave the way for subsequent signal
processing. This technique is widely used in many fields, such
as 5G, Internet of things and electronic warfare, etc. With the
increasing complexity of wireless communication scenarios,
there are more and more influencing factors on modulation
classification algorithm, which leads to the failure of the
recognition algorithm in a single scene. Therefore, it is very
important to explore modulation classification algorithm with
strong adaptability to the environment [1], [2].
It is found that the modulation classification algorithm
is mainly divided into two parts: automatic classification
algorithm based on Deep Learning and manual calculation
classification algorithm [3]–[6].
The manual calculation classification algorithm is mainly
divided into two parts: likelihood ratio based (LrB) method
and on the feature extraction based (FEB) method. The
LrB method mainly calculates and compares the likelihood
The associate editor coordinating the review of this manuscript and
approving it for publication was Zhaojun Li . received signal to generate datasets, and then sends the data
classification accuracy rate. The LrB method has a high
classification accuracy rate, but the cost is also high. The
likelihood function of the LrB method is complex, the calcu-
lation is large, and some communication systems are difficult
to bear. The adaptability of the LrB method is poor, a lot of
prior knowledge is required, and the parameters are derived
under certain conditions and are only applicable to a specific
scene. The FEB method has a very obvious improvement in
computational complexity, and the signal features are easy
to extract. In the case of less prior knowledge, the param-
eters extracted by the FEB method are matched with the
machine learning classifier, and the better recognition perfor-
mance can be obtained. The FEB method, however, is usually
based on a specific signal sample to extract features and
set the decision method, thus being affected by the channel
environment [7]–[11].
In recent years, with the development of Deep Learning
(DL), the application of Deep Learning in the physical layer
also appears. On the basis of physical layer transmission, it is
a hot topic to realize the modulation classification technique
by using Deep Learning. The DL method first converts the

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 9, 2021 26313
 Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
set to the DL network for automatic classification to obtain
the classification result [12]. The Deep Learning is applied to
the modulation classification of the signals, with the follow-
ing three advantages:
• Deep Learning method avoids the manual feature extrac-
tion and selection, and reduces the sensitivity of the
feature to the environment.
• Deep Learning model is data-driven, requires a large
number of data training models, and the communication
signals can be easily acquired through the equipment,
the input conditions of the deep learning network can be
satisfied.
• The method of Deep Learning is in accordance with
the current research trend, and has more exploration
value
According to the above description, the automatic modula-
tion classification method based on DL is used in this paper.
In addition to the algorithm, many scholars have studied the
influence of different noise on the modulation classification,
mainly including the Gaussian noise and the non-Gaussian
noise. Gaussian white noise is often used to simulate the
ideal noise environment, and the influence on signal classi-
fication is relatively small. With the increasing complexity of
the channel environment, Gaussian white noise is not close
enough to the characteristics of the channel, which leads to
the degradation of the performance of the classification model
established in the simulation of Gaussian noise. Consider-
ing the deterioration of the channel, the algorithm of using
non-Gaussian noise to simulate the actual noise environment
follows one after another. When dealing with non-Gaussian
white noise, there are two methods, one is to whiten the
color noise, and the other is to define the non-Gaussian noise
as a kind of generalized Gaussian distribution, Alpha stable
distribution noise, which is suitable for describing the noise of
communication channel. Almost all the proposed algorithms
are only in a single noise environment, but the single noise
setting has some limitations. In the actual communication
process, there are many uncertain factors, which can bring
challenges to the decision-making of the communication
system. The algorithms proposed in this paper are suitable
for Gaussian noise and non-Gaussian noise. No matter what
kind of noise in the input signal is closer to, the proposed
algorithm has good performance and strong inclusiveness to
noise [13], [14].
The signal-to-noise ratio is usually used to measure the
relationship between the pure signal and the noise in the
received signal, and the signal-to-noise ratio has a certain
influence on the signal classification. At present, an algorithm
based on DL method is used for signal classification, and
each dataset under each signal-to-noise ratio is sent to a
DL network to train a model, and a plurality of models are
trained in a plurality of signal-to-noise ratios. The resulting
classification results will be more convincing if the datasets at
all signal-to-noise ratios are fed to the deep learning network
at the same time [15]–[17].
26314
In this paper, two signal conversion methods are proposed
to avoid the problem of unequal information quality in collab-
orative computing for AMC. Automatic modulation classifi-
cation method of DL is used, and big data sample driven DL
model is used to get better classification results. In the process
of signal generation, the simulated noise includes Gaussian
noise and non-Gaussian noise, and the noise with different
non-Gaussian degrees is taken into account. All the datasets
under MSNRs are sent to the network together for classifi-
cation. The proposed method can get better performance in
both Gaussian noise and non-Gaussian noise
The structure of the rest of this paper is shown below.
In Section II mainly describes the existing algorithms which
are similar to those proposed in this paper. In Section III
mainly describes the establishment of the system model,
including signal model, noise model and classification
network. In Section IV mainly describes the algorithm
framework, including the existing problems and improved
algorithms. Section V is the simulation experiment of the
algorithm. Section VI is the summary of the full paper.
II. RELATED WORKS
In recent years, deep learning has been applied to the paper of
modulation classification. In [7], Peng et al proposed for the
first time that the signal domain be converted into a graph
domain, the geometric relation of the constellation map is
used to recognize the modulation type, and the constellation
map of different signals is constructed to make it have dif-
ferent characteristics. In this paper, the influencing factors
of signal generation constellation map are studied, including
the conversion method of data, the size of picture selection
and the pixel size of picture, and the recognition performance
is analyzed under different deep learning networks. On the
basis of the transformation method of constellation map pro-
posed in this paper, several subsequent authors have improved
the algorithm and structure. In [8], a point density method
is proposed to color the constellation maps. Tu Ya, et al.
proposed a new framework to enhance the data using the
auxiliary classifier genetic adversarial network, and use the
CNN model to classify the data, which can gain 0.1%∼6%
increase in accuracy. In this paper, some training tricks are
used to solve the problems of over-fitting, non-convergence
and model collapse. In view of the fact that the model needs a
lot of training data to avoid the problem of over-fitting, in [9],
Tu Ya, et al. proposed the application of semi-supervised
learning to unlabeled data to reduce the problem of over-
fitting. In [10], Wang Yu et al. proposed a method of com-
bining double-layer CNN, in which the first layer of CNN
is trained with the real part sequence and the imaginary part
sequence of the signal, and the other layer is trained with
the transformed constellation map. This method is easier to
implement in actual projects than the full use of graphs for
training.
Considering that the length of the signal is often larger
than the length of the deep learning network, and the ran-
dom burst pulse is easy to appear in the signal, in [11],
VOLUME 9, 2021
Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
Zheng et al proposed a fusion classification method to seg-
ment the signal and send the multi-segment signal into the
network. The fusion methods include the voting fusion of
multi-stage results, the average fusion of confidence and the
average fusion of features, and the performance of the three
methods is compared. Considering the uncertainty of noise
types in the channel, in [12], Hu et al proposed a robust
method for modulation classification under uncertain noise.
Because of the obvious characteristics of constellation, our
paper also adopts the data conversion format of constella-
tion, improves the processing, and finally realizes modulation
classification.
III. SYSTEM MODEL
A. SYSTEM MODEL
In non-cooperative communication, the prior information of
the signal is not known to the receiver. The steps of tradi-
tional modulation classification algorithm are shown in Fig.1,
mainly including signal preprocessing, feature extraction
and classification. In addition, to match the classification
model, it is necessary to estimate the SNR of the current
channel [18]–[20].
FIGURE 1. Traditional modulation classification steps.
The modulation classification algorithm in the traditional
method has complicated operation steps and tedious model
training. The method proposed in this paper is shown
in Fig. 2. The received signals are preprocessed and sent to
the neural network, and the network automatically extracts
features and classifies. The proposed algorithm avoids the
redundancy of SNR estimation and only needs to train
a model, which greatly reduces the complexity of the
algorithm [21]–[23].
FIGURE 2. The proposed modulation classification steps.
B. SIGNAL MODEL
Typically, the received signal is represented as
x(t) = s(t) + n(t)
The signals used in this paper are digital signals, where
x(t) and s(t) represent the received signal and the transmit-
ted signal, respectively, and n(t) is the noise in the analog
environment.n(t) include Gaussian noise and non-Gaussian
noise [24]–[27].
VOLUME 9, 2021
C. SIGNAL MODEL
Alpha stable distribution (ASD) is a generalized form
of Gaussian distribution, and it is also one of the main
error sources in practical wireless communication systems.
Because of its stability in probability distribution and peak
pulse characteristics, it can accurately describe the statistical
characteristics of noise in the actual wireless communication
environment.
The eigenfunction of ASD can be expressed as
φ(θ ) = exp jµθ − γ |θ |α [1 + jζ sgn(θ)ω(θ, α)]

(2)
where
(
tan(π α/2), α 6= 1
ω(θ, α) = , α(0 < α ≤ 2)
(2/π) log |θ |, α = 1
is characteristic index, which represents the degree of concen-
tration of stable distribution. The smaller the value of α is,
the more concentrated the corresponding stable distribution
is, that is, the stronger the pulse characteristic is. ξ (−1 ≤
ξ ≤ 1) is a skew parameter, which measures whether the
stability distribution has a symmetrical characteristic. µ is
the position parameter used to determine the position of the
stable distribution. γ ≥ 0 is a scale parameter, which is used
to measure the width of a stable distribution. And sgn(·) is a
symbolic function, defined as:

 1,
 θ >0
sgn(θ) = 0, θ =0 (3)
−1, θ < 0


Because the second-order statistical moment of Alpha sta-
ble distribution noise does not exist and cannot represent its
variance, the signal-to-noise ratio (SNR) in the traditional
sense is not fully in line with Alpha stable distribution noise,
so the mixed signal-to-noise ratio (MSNR) is defined as
follows:
δs2
MSNR=10 log10 ( ) (4)
γ
where γ ≥ 0 is a scale parameter, δs2 is the variance of signal.
It is worth noting that the alpha-stable distribution can be
seen as a generalization of the Gaussian distribution, which
is of a particular nature itself. When α = 2, the alpha-stable
distribution is degraded to Gaussian distribution. Therefore,
the α value we set in this paper includes some sampling points
between 0 and 2. When α = 2, it is considered that the
current noise is Gaussian white noise. When α 6 = 2, it is
considered that the current noise is non-Gaussian noise. The
smaller α means the stronger the non-Gaussian characteristic,
(1) we continue to study the classification performance of the
system under different intensities of non-Gaussian noise [28].
D. DEEP LEARNING NETWORK
The essence of deep learning is to learn more useful fea-
tures by constructing machine learning models with many
hidden layers and massive training data, so as to improve
26315
 Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
the accuracy of classification or prediction. Therefore, ‘deep
model’ is a means, ‘feature learning’ is an end. Deep Learn-
ing emphasizes the depth of model structure and highlights
the importance of feature learning. Through layer-by-layer
feature transformation, the feature representation of samples
in the original space is transformed into a new feature space,
which makes it easier to classify or predict. Compared with
the method of constructing features by artificial rules, big data
is more able to depict the rich internal information of data by
using big data to learn features.
Convolution neural network is one of the feedforward
neural networks. It has a unique effect on image process-
ing, including at least convolution layer and pool layer in
structure. Convolution neural network is a deep learning
network which has been developed in recent years, and has
been widely paid attention to and applied in enterprises.
Representative convolution neural networks include AlexNet,
VGG, LeNet and so on. In this paper, CNN is used to solve
not complete the modulation classification problem. Because
our dataset picture content is relatively simple, we use the
classical CNN model-AlexNet.
AlexNet carries forward the idea of LeNet and applies the
basic principle of CNN to a very deep and wide network.
AlexNet has several features as shown below
• Using the ReLU activation function, the deep convolu-
tion network based on ReLU is several times faster than
the network training based on tanh and sigmoid.
• In AlexNet, Dropout, is mainly used in the last few full
connection layers to randomly ignore some neurons by
using Dropout in training, which can avoid overfitting of
the model.
• AlexNet all uses maximum pooling to avoid the fuzzi-
fication effect of average pooling. Moreover, it is pro-
posed in AlexNet that the concession length is smaller
than the size of the pooled kernel, so that there will be
overlap and coverage between the outputs of the pooling
layer, which improves the richness of the features.
IV. MODULATION CLASSIFICATION SCHEME
The modulation classification framework based on deep
learning is shown in Fig. 2, the received signal is converted
into a picture after signal processing and sent to the CNN
network for automatic classification.
In the process of converting signals into diagrams, there
will be interference problem, so in this paper mainly improves
the algorithm in the signal processing stage.
A. DATA CONVERSION
Constellation map is the two-dimensional geometric charac-
teristics of the signal. In this paper, the complex signal is used
to generate the constellation map. It is worth noting that the
ASK signal is a complex signal with imaginary part 0, and it
still has two-dimensional characteristics after superposition
of complex noise.
26316
The constellation map is colored by point density method.
And the point density of the i − th symbol in each sample ρ(i)
is calculated as follows:
L
f [m(i) − m(j)| < r&&|n(i) − n(j)| < r]
P
j=1
ρ(i) = (5)
L
where m(i) obtain the horizontal axis and n(i) obtain the
vertical axis value of the i − th symbol, L is the number of the
symbols of sample, r is half the length of the selected square
region when calculating the point density.
B. EXISTING PROBLEM
In real environment, the noise often has the peak and the
pulse burst property. The most remarkable characteristic of
this kind of noise is that its algebra tail is thicker than the
exponential tail of Gaussian distribution, and the amplitude
of its probability density function is obviously higher than
that of Gaussian distribution. Therefore, the single Gaussian
distribution model cannot fully describe this kind of impulse
noise. Theoretical studies have shown that the special proper-
ties of Alpha-stable distribution noise can effectively describe
the noise present in the real situation. The scatter diagram
of Alpha-stable distribution noise is shown in Fig. 3, where
we can see the sharp noise points circled by the red box.
Since the Alpha-stable distribution noise does not have the
statistical characteristics of the second order and high order,
the conventional method for extracting the time-frequency
characteristic and the statistical feature is not suitable for the
noise.
FIGURE 3. The scatter diagram of alpha-stable distribution noise.
In [7], Tu et.al, proposes a modulation recognition method
based on Contour Stella Image in Gaussian white noise
environment. We replace the Gaussian white noise with the
Alpha-stable distribution noise, the concentrated part of the
graph is affected by noise, the irregular beating occurs, and
the original geometric characteristics have been lost, so the
constellation point of QPSK signal cannot be seen.
The sharp pulse of Alpha-stable distribution noise has a
great influence on the feature extraction and classification
VOLUME 9, 2021
Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
of signal constellation map, so our main task in this paper
is to preprocess the sharp pulse before the signal domain is
converted into graph domain, and to suppress or remove the
sharp pulse.
C. IMPROVEMENT
Because the noise has serious trailing and sharp pulse, and the
amplitude value of the signal is in the stable range, we pro-
pose a method which can not only suppress the sharp noise
point, but also improve the sensitivity of the weak value in a
certain range. Logarithm is a common statistical method, its
main function is to weaken the large value point according to
a certain base number, at the same time, it can also improve
the sensitivity of the weak value, which can just solve the
current problem.
Therefore, the logarithmic operation of the signal sequence
is carried out in this paper, and the constellation map is gener-
ated. The original constellation map is mapped to logarithmic
domain to form logarithmic constellation map.
1) DIRECT LOGARITHM
The first improved method is to perform logarithmic opera-
tion directly on the received complex sequences, we call this
method the direct logarithm (D-Log) method. Assuming that
the received complex signal is xn = an + bn i, can also be
represented as
q Similarly, we do the following operations on bn sequences.
an + bn i = a2n + b2n (cos θn + i sin θn ) (6)
an bn
where cosθn = p and sinθn = p
a2n + b2n a2n + b2n
According to Euler formula
eiθn = cos θn + i sin θn
Then, do natural logarithmic operations on xn , we can get
the following results:
q
ln(an + bn i) = ln[ a2n + b2n (cos θn + i sin θn )]
q
= ln( a2n + b2n ) + ln[(cos θn + i sin θn )]
= An + θn i
where An is real number.
Because θ can add countless cycles, there are countless
solutions in theory, but only the angle values in the unit cycle
are usually selected in the mathematical operation.
Then according to the base changing formula of logarithm,
the logarithmic values of any base number can be obtained.
logn (X )
logm (X ) =
logn m
where m is the base number that you want to change, and n
is the known base number. In this paper, in order to suppress
the sharp value of noise to a large extent, set m = 10.
VOLUME 9, 2021
2) REAL-IMAGE LOGARITHM
The first improved method of direct logarithm changes the
geometric characteristics of constellations and reduces the
divisibility of signals to a certain extent. The second improved
method is still to use the idea of logarithm to improve, but
the real part and the imaginary part of the complex signal
are logarithmic respectively, named Real-Image logarithm
method (RI-Log). Assumption xn = an + bn i, an represents
the real part of the signal and bn represents the imaginary part
of the signal.
Because there are negative numbers in both an and bn , and
the logarithms of negative numbers are complex. Therefore,
we make a little improvement on the basis of real logarithm,
the logarithmic operation of the real part and the imaginary
part of the complex signal is carried out respectively. If a
logarithmic operation is directly performed on the I and Q
channel sequences, a sharp pulse will still occur when a
pulse approaching 0 is present, not in accordance with the
purpose of which we have been a sharp pulse. So we fine-tune
the logarithmic algorithm, so that the logarithmic operation
will not produce sharp values, and can achieve the desired
purpose.
We do the following operations on an sequence.
 log10 (a + 1), a > 0
 
an = 0, a=0 (10)
−log10 (−a + 1), a < 0
 
 log10 (b + 1), b > 0
 
bn = 0, b=0 (11)
. −log10 (−b + 1), b < 0
 
V. EXPERIMENTS
In this part, we carry on the simulation experiments to D-Log
(7)
method and RI-Log method respectively. The experiments
mainly include three contents: (1) Classification results under
unknown MSNR; (2) Classification results under different α
conditions, that is, classification results under Gaussian noise
and non-Gaussian noise. Among them, α = 2 is a simulated
Gaussian noise environment, and other parameters represent
noise with different non-Gaussian degrees. (3) Compared
with the traditional modulation recognition method based on
(8) feature extraction.
At the end of this section, the proposed method is compared
with existing algorithms.
A. DATASET CONSTRUCTION
In this paper, MATLAB is used to generate signals and
simulate datasets, its own Deep Learning Network toolbox
is used to classify datasets, and generate confusion matrix
and accuracy rate curve of classification results. Here, we do
(9) some simulation experiments to verify the effectiveness of
the improved algorithm. In the experiments, the signals used
include 2ASK, 4ASK, 8PSK, QPSK, OQPSK, 16QAM and
64QAM. The real and imaginary parts of each received signal
include 10,000 sampling points, which are used to generate
26317
 Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
images. Each signal generates 500 simulation images under
the same MSNR or the same α value, these images were put
into AlexNet, 70% for training and 30% for testing. As shown
in Fig.4, we built a data set that contained a large amount of
data. The data set is labeled with each signal type, where each
signal contains data under multiple SNR. The same approach
is used for the construction of the dataset for α.
FIGURE 4. Dataset construction.
B. TRAINING PATTERN
As shown in Fig.5, in the traditional modulation classification
method, no matter which classifier is used, a model is trained
under each SNR. This means that multiple iterations of the
model training are required.
FIGURE 5. Training pattern of traditional modulation classification
method.
At the same time, the current MSNR needs to be estimated
in order to match the model in the verification process. It can
be seen that this method is not only very tedious in operation
process, but also redundant in model training.
As shown in Fig.6 the constructed mixed MSNR in our
proposed algorithm is used as the data set and input into
the network at the same time. Only one model training is
required. The trained model can be applied to multiple SNR
environments. At the same time, this method avoids the step
of SNR estimation.
C. EXPERIMENTS UNDER UNKNOWN MSNRs
1) EXPERIMENTS OF D-LOG METHOD
UNDER UNKNOWN MSNRS
The experiments under D-Log method include 16QAM,
BPSK, QPSK, OQPSK and 8PSK signals. The range of
26318
FIGURE 6. Training pattern of the proposed modulation classification
method.
MSNR is 0 to 16 dB, and the step is 2. Set the value of
another influence parameter α to 1.6. Each signal generates
500 image datasets under different MSNRs. The datasets of
the same signal under different MSNRs are regarded as the
same label, and the different signals are used as different
labels. In which 70% of the data is used for the training model
and 20% of the data is used to test the results, 10% of the data
is validation set.
In general, that data of different MSNRs are respectively
trained to obtain the classification result of test data, so that
the universality is lack to a certain extent.
In this experiment, the datasets of different signals are
trained at the same time under different MSNRs, and only
one model needs to be trained. In the MSNR range from 0 to
16dB, a good classification effect can be obtained by using
the model under unknown MSNRs.
The model training process under the D-Log method is
shown in Fig. 7, the blue line represents the change curve of
loss value, corresponding to the left Y-axis. The red line is the
change curve of accuracy, corresponding to the right Y-axis.
It can be seen from Fig.7 that, under the D-log method,
the accuracy of the validation set fluctuated with the decline
of the loss value of Batch data. After reaching 9 training
cycles, the training results all tended to be stable, and the
lower the loss value in the stability stage, the higher the
accuracy of the validation set.
FIGURE 7. Model training under D-Log method.
As shown in Fig. 8, it is the classification result of D-Log
method under unknown MSNRs. It can be seen that under
the unknown MSNRs, the recognition effect can still reach
more than 90%. It can be seen that this method still has good
classification performance when unknown MSNRs.
VOLUME 9, 2021
Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment

8PSK signals. Here, we add the ASK signal. Because in the

I/Q sequence of ASK, the Q sequence value is 0. Using the
D-log method to perform a complex logarithm operation on
a single sequence does not work well. The range of MSNR
is also 0 to 16 dB, and with a 2 dB step. Set the value of
another influence parameter α to 1.6. Each signal generates
500 image datasets under different MSNRs. The datasets of
the same signal under different MSNRs are regarded as the
same label, and the different signals are used as different
labels. Only one model need be trained similarly.
The model training process under the RI-Log method is
shown in Fig.10, After reaching 8 training cycles, the training
results all tended to be stable, and the lower the loss value in
the stability stage, the higher the accuracy of the validation
FIGURE 8. Classification result of D-Log method under unknown MSNRs.
set. Compared with Fig.7, this method converges faster and
the average recognition accuracy of the validation set is better.

FIGURE 10. Model training under RI-Log method.

FIGURE 9. Test results of different MSNRs in D-Log trained model.

As shown in Fig. 11, it is the classification result of RI-Log
method under unknown MSNRs. It can be seen that under
the unknown MSNRs, the classification accuracy of 16QAM,
Then, the data under different MSNRs are sent to the
2ASK, 4ASK, BPSK, 8PSK signals can reach more than
trained model for testing respectively. The test classification
80%. The accuracy of OQPSK under unknown MSNRs can
results of D-Log method under different MSNRs are shown
reach 75.9%. It can be seen that this method still has good
in Fig. 9. It can be seen that QPSK, OQPSK and 8PSK have
classification performance when unknown MSNRs.
poor classification performance at lower MSNRs, and the
accuracy of 5 types of signals can reach more than 85% over
4 dB.
The prediction classification results of the hybrid MSNRs
training model and the test results under different MSNRs
show that the D-Log method has very good classification
performance. The signal modulation type with high accuracy
can be obtained in the environment of unknown MSNRs.
The D-Log method changes the geometric structure of
constellation to a certain extent, and the geometric structure
has certain physical meaning. In order to ensure the geometric
properties and physical meaning of constellation map, the
RI-Log method reduces the range of constellation map and
does not change the geometric structure of constellation map.
Therefore, this paper carries on the simulation verification to
the RI-Log method.
FIGURE 11. Classification result of RI-Log method under unknown
MSNRs.
2) EXPERIMENTS OF RI-LOG METHOD
UNDER UNKNOWN MSNRS At present, the training models with different signal-to-
The experiments under RI-Log method contain 6 types sig- noise ratio (SNR) have been obtained, and several different
nals, including 16QAM, 2ASK, 4ASK, BPSK, OQPSK and MSNR data have been selected to test the trained model.

VOLUME 9, 2021 26319

 Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment

In this part, seven MSNR test points between 0 and 12dB
are selected. The test classification results of RI-Log method
under different MSNRs are shown in Fig. 10. It can be seen
that BPSK, OQPSK and 4ASK have poor classification per-
formance at lower MSNRs, and the accuracy of 6 types of
signals can reach more than 85% over 4 dB.
The RI-Log method only reduces the range of constella-
tions, preserves the geometric structure of constellations, and
has more physical interpretable significance. As can be seen
from Fig. 11 and Fig. 12, the RI-Log method still has good
classification performance.
In this part, the simulation of the two methods in Gaussian
and non-Gaussian noise environment is carried out to verify
whether the two methods can be effective under different
noise and different non-Gaussian degrees.
1) EXPERIMENTS OF D-LOG METHOD UNDER DIFFERENT α
When α is less than 2, the selected range is 0.3 to 1.8, and the
step is 0.3. Each signal generates 500 image datasets under
different α. Set the value of MSNR to 10dB.
As shown in Fig. 13, as a result of the classification of
the D-Log method at different α, it can be seen from the
figure that the five signals have good performance in sup-
pressing non-Gaussian noise, and are suitable for Gaussian
noise and non-Gaussian noise. When MSNR is equal to 10dB,
when α is greater than 0.3, the accuracy of signal classifica-
tion can reach more than 95%.

FIGURE 12. Test results of different MSNRs in RI-Log trained model.

In conclusion, both methods can suppress the interference

of sharp value of Alpha stable distribution noise to modu-
lation classification, and the two methods have good mod-
ulation classification performance in the case of unknown FIGURE 13. Classification result of D-Log method under unknown α.
MSNRs. The former method has obvious characteristics and
can obtain higher recognition accuracy, but it changes the
2) EXPERIMENTS OF RI-LOG METHOD UNDER DIFFERENT α
geometric structure of constellation map, the latter preserves
When α is less than 2, the selected range is 0.4 to 1.8, and the
the physical structure of constellation map, and can also
step is 0.2. Each signal generates 500 image datasets under
obtain better classification effect.
different α. Set the value of MSNR to 10dB.
D. EXPERIMENTS UNDER DIFFERENT α As shown in Fig. 14, as a result of the classification of
In the process of modulation classification, in addition the RI-Log method at different α, it can be seen from the
to considering the influence of the signal-to-noise ratio,
the influence of the α parameter in the classification is also
considered. α = 2 means that the current noise is Gaussian
noise, α < 2 means non-Gaussian noise, and the smaller α
means the greater the degree of non-Gaussian noise.
At first, we tried to realize the simulation verification at the
unknown α value, and the signal data sets corresponding to
the different α values are put together to train a model so as
to have the universality of the α value. However, due to the
obvious difference of signal data features corresponding to
different α values, it is difficult to find its common features in
the process of training, which leads to the poor classification
effect. Therefore, the classification simulation experiment
of different α value is carried out by using the traditional
method, that is, each α value is used for training a model
respectively. FIGURE 14. Classification result of RI-Log method under unknown α.

26320 VOLUME 9, 2021

Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
figure that the six signals have good performance in suppress-
ing non-Gaussian noise and are suitable for Gaussian noise
and non-Gaussian noise. For 8PSK signals, when A is less
than 0.6, the signal classification accuracy is low, and it can be
achieved 90 percent signal classification accuracy is greater
than 0.6 at MSNR is equal to 10dB. For another five types
signals, when α is greater than 0.4, the accuracy of signal
classification can reach more than 95%.
In conclusion, both methods suppress the non-Gaussian
noise performance, and are both suitable for Gaussian noise
and non-Gaussian noise, and have strong robustness. In that
latter method, the accuracy of the individual signal identifica-
tion may be low when the non-Gaussian degree is high. But
the overall classification performance of both methods is very
good.
E. COMPARISON EXPERIMENTS
The traditional modulation classification method is based on
feature extraction. Feature parameters are extracted manually
and appropriate classifier is selected to realize modulation
type classification. In this comparison algorithm, the features
of fractional low-order cyclic spectrum (FLCS) and higher
order cumulant are selected.
The average recognition rate of the two methods based
on eigenvalue extraction and the two algorithms proposed in
this paper is shown in Fig.15. The blue lines in Fig.15 had
a significantly lower recognition rate than the other three
algorithms. It shows that the high-order cumulant feature
is not suitable for the recognition problem under α stable
distributed noise. In the other three methods, when MSNR
is greater than or equal to 2dB, the recognition rate of RI-Log
and D-Log is better than that of FLCS. The D-Log method
has the highest recognition rate. Through this experiment,
the effectiveness of the two methods proposed in this paper is
proved.
FIGURE 15. Compare the recognition rate of different methods.
F. ANALYSIS
The method proposed in this paper has the following
advantages.
VOLUME 9, 2021
1) In terms of recognition performance, the two algo-
rithms proposed in this paper are superior to the tra-
ditional recognition methods when the SNR is greater
than 2dB.
2) In terms of robustness, the method proposed in this
paper can be artificially independent and self-adaptive
within a certain range of MSNR, and can be applied to
both Gaussian noise and non-Gaussian noise.
3) In terms of model training, the mixed data set con-
structed in this paper only needs to train one model,
which reduces the complex operation in model training.
However, this paper still has the following shortcomings.
1) The randomly selected test sets after model training
will also have a certain impact on the classification
results. For example, the improved constellation map
of 16QAM signal presents 4 constellation points at
low MSNR, 16 constellation points at high MSNR,
and QPSK signal is also 4 constellation points. if the
randomly selected test set contains more 16QAM data
at low MSNR, it may appear that it is recognized
as QPSK signal, which leads to poor classification
results.
2) In general, if there are N SNR sampling points, we need
to train the model N times. and this paper only needs
to train one model, which reduces the complexity to a
certain extent. However, the training time of a single
model is longer than that of multiple models.
These issues are also the direction of future improvements.
VI. CONCLUSION
In this paper, two methods of logarithmic constellation are
proposed, direct logarithm method and real-image logarithm
method. The simulation results show that the two methods
can be used in both Gaussian noise and non-Gaussian noise
environments. Compared with the traditional model training
method under each MSNRs, it is more universal and extensive
to send the data from different MSNRs to the network to train
a model at the same time. We use different α to simulate
different noises. α equal to 2 is Gaussian noise and α less
than 2 is non-Gaussian noise. The simulation results show
that the two algorithms have good classification performance
under unknown MSNRs, and the classification performance
is still good under different α.
REFERENCES
[1] O. Dobre, ‘‘Signal identification for emerging intelligent radios: Classical
problems and new challenges,’’ IEEE Instrum. Meas. Mag., vol. 18, no. 2,
pp. 11–18, Apr. 2015.
[2] Y. Lin, Y. Tu, Z. Dou, L. Chen, and S. Mao, ‘‘Contour stella
image and deep learning for signal recognition in the physical layer,’’
IEEE Trans. Cogn. Commun. Netw., early access, Sep. 18, 2020, doi:
10.1109/TCCN.2020.30246.10.
[3] T. O’Shea and J. Hoydis, ‘‘An introduction to deep learning for the physical
layer,’’ IEEE Trans. Cognit. Commun. Netw., vol. 3, no. 4, pp. 563–575,
Dec. 2017.
[4] L. Han, F. Gao, Z. Li, and O. A. Dobre, ‘‘Low complexity automatic
modulation classification based on order-statistics,’’ IEEE Trans. Wireless
Commun., vol. 16, no. 1, pp. 400–411, Jan. 2017.
26321
 Z. Cai et al.: Performance Evaluation of Big Data-Driven Modulation Classification in Complex Environment
[5] J. Zhang, D. Cabric, F. Wang, and Z. Zhong, ‘‘Cooperative mod-
ulation classification for multipath fading channels via expectation-
maximization,’’ IEEE Trans. Wireless Commun., vol. 16, no. 10,
pp. 6698–6711, Oct. 2017.
[6] F. Meng, P. Chen, L. Wu, and X. Wang, ‘‘Automatic modulation classi-
fication: A deep learning enabled approach,’’ IEEE Trans. Veh. Technol.,
vol. 67, no. 11, pp. 10760–10772, Nov. 2018.
[7] S. Peng, H. Jiang, H. Wang, H. Alwageed, Y. Zhou, M. M. Sebdani, and
Y.-D. Yao, ‘‘Modulation classification based on signal constellation dia-
grams and deep learning,’’ IEEE Trans. Neural Netw. Learn. Syst., vol. 30,
no. 3, pp. 718–727, Mar. 2019.
[8] Y. Tu, Y. Lin, J. Wang, and J.-U. Kim, ‘‘Semi-supervised learning with gen-
erative adversarial networks on digital signal modulation classification,’’
Comput. Mater. Continua, vol. 55, no. 2, pp. 243–254, 2018.
[9] Y. Lin, M. Wang, X. Zhou, G. Ding, and S. Mao, ‘‘Dynamic spectrum
interaction of UAV flight formation communication with priority: A deep
reinforcement learning approach,’’ IEEE Trans. Cognit. Commun. Netw.,
vol. 6, no. 3, pp. 892–903, Sep. 2020.
[10] Y. Wang, M. Liu, J. Yang, and G. Gui, ‘‘Data-driven deep learning for
automatic modulation recognition in cognitive radios,’’ IEEE Trans. Veh.
Technol., vol. 68, no. 4, pp. 4074–4077, Apr. 2019.
[11] S. Zheng, P. Qi, S. Chen, and X. Yang, ‘‘Fusion methods for
CNN-based automatic modulation classification,’’ IEEE Access, vol. 7,
pp. 66496–66504, 2019.
[12] S. Hu, Y. Pei, P. P. Liang, and Y.-C. Liang, ‘‘Deep neural network for robust
modulation classification under uncertain noise conditions,’’ IEEE Trans.
Veh. Technol., vol. 69, no. 1, pp. 564–577, Jan. 2020.
[13] J. Ma and T. Qiu, ‘‘Automatic modulation classification using cyclic
correntropy spectrum in impulsive noise,’’ IEEE Wireless Commun. Lett.,
vol. 8, no. 2, pp. 440–443, Apr. 2019.
[14] Y. Tu, Y. Lin, C. Hou, and S. Mao, ‘‘Complex-valued networks for auto-
matic modulation classification,’’ IEEE Trans. Veh. Technol., vol. 69, no. 9,
pp. 10085–10089, Sep. 2020.
[15] Y. Wang, J. Wang, W. Zhang, J. Yang, and G. Gui, ‘‘Deep learning-based
cooperative automatic modulation classification method for MIMO sys-
tems,’’ IEEE Trans. Veh. Technol., vol. 69, no. 4, pp. 4575–4579, Apr. 2020.
[16] S. Huang, Y. Jiang, Y. Gao, Z. Feng, and P. Zhang, ‘‘Automatic modulation
classification using contrastive fully convolutional network,’’ IEEE Wire-
less Commun. Lett., vol. 8, no. 4, pp. 1044–1047, Aug. 2019.
[17] Y. Wang, J. Yang, M. Liu, and G. Gui, ‘‘LightAMC: Lightweight automatic
modulation classification via deep learning and compressive sensing,’’
IEEE Trans. Veh. Technol., vol. 69, no. 3, pp. 3491–3495, Mar. 2020.
[18] Z. Zhang, X. Guo, and Y. Lin, ‘‘Trust management method of D2D com-
munication based on RF fingerprint identification,’’ IEEE Access, vol. 6,
pp. 66082–66087, 2018.
[19] X. Yan, G. Zhang, and H.-C. Wu, ‘‘A novel automatic modulation classifier
using graph-based constellation analysis for M -ary QAM,’’ IEEE Com-
mun. Lett., vol. 23, no. 2, pp. 298–301, Feb. 2019.
[20] D. Wang, M. Zhang, Z. Li, J. Li, M. Fu, Y. Cui, and X. Chen, ‘‘Modu-
lation format recognition and OSNR estimation using CNN-based deep
learning,’’ IEEE Photon. Technol. Lett., vol. 29, no. 19, pp. 1667–1670,
Oct. 1, 2017.
[21] M. H. Shah and X. Dang, ‘‘Robust approach for AMC in frequency selec-
tive fading scenarios using unsupervised sparse-autoencoder-based deep
neural network,’’ IET Commun., vol. 13, no. 4, pp. 423–432, Mar. 2019.
[22] M. Sadeghi and E. G. Larsson, ‘‘Adversarial attacks on deep-learning based
radio signal classification,’’ IEEE Wireless Commun. Lett., vol. 8, no. 1,
pp. 213–216, Feb. 2019.
[23] R. Li, L. Li, S. Yang, and S. Li, ‘‘Robust automated VHF modulation
recognition based on deep convolutional neural networks,’’ IEEE Commun.
Lett., vol. 22, no. 5, pp. 946–949, May 2018.
26322
[24] Y. Lin, Y. Tu, and Z. Dou, ‘‘An improved neural network pruning technol-
ogy for automatic modulation classification in edge devices,’’ IEEE Trans.
Veh. Technol., vol. 69, no. 5, pp. 5703–5706, May 2020.
[25] S. Chen, S. Zheng, L. Yang, and X. Yang, ‘‘Deep learning for large-scale
real-world ACARS and ADS-B radio signal classification,’’ IEEE Access,
vol. 7, pp. 89256–89264, 2019.
[26] Y. Lin, H. Zhao, Y. Tu, S. Mao, and Z. Dou, ‘‘Threats of adversarial attacks
in DNN-based modulation recognition,’’ in Proc. IEEE Conf. Comput.
Commun. (INFOCOM), Jul. 2020, pp. 2469–2478.
[27] Y. Lin, X. Zhu, Z. Zheng, Z. Dou, and R. Zhou, ‘‘The individual iden-
tification method of wireless device based on dimensionality reduction
and machine learning,’’ J. Supercomput., vol. 75, no. 6, pp. 3010–3027,
Jun. 2019.
[28] M. Ma, Z. Li, Y. Lin, L. Chen, and S. Wang, ‘‘Modulation classification
method based on deep learning under non-Gaussian noise,’’ in Proc. IEEE
91st Veh. Technol. Conf. (VTC-Spring), Antwerp, Belgium, May 2020,
pp. 1–5.
ZHUORAN CAI received the Ph.D. degree in
information and communication engineering from
the Harbin Institute of Technology, in 2013. He is
currently an Associate Professor with the School
of Opto-Electronic Information Science and Tech-
nology, Yantai University, Yantai, China. His
research interests include deep learning methods,
cognitive radio, and compressed sensing.
JIDONG WANG received the bachelor’s degree
from Harbin Engineering University, China,
in 2016, and the master’s degree from the Wuhan
Maritime Communication Research Institute,
China, in 2019.
Her research interests include digital signal
processing and HF communication.
MINGHUAN MA received the B.S. degree in
electronic engineering from Heilongjiang Univer-
sity, China, in 2018. She is currently pursuing the
M.S. degree with the College of Information and
Communication Engineering, Harbin Engineering
University, China.
Her research interests include deep learning for
modulation recognition and signal analysis, and
big data analytics of wireless spectrum.
VOLUME 9, 2021