FEATURE ARTICLE: IT SECURITY-VULNERABILITY DETECTION

Malware Detection and Classification

Based on Graph Convolutional Networks
and Function Call Graphs
Hsiang-Yu Chuang, Jiann-Liang Chen , and Yi-Wei Ma , National Taiwan University of Science and
Technology, Taipei, 106335,Taiwan

New types and variants of malware are constantly and rapidly being developed.
Identifying malware effectively and quickly has become a primary goal of information
security analysts. This study proposes a malware detection and classification model
that is based on graphical convolutional networks and function call graphs. Analyzing
the behavior of malware executions through sandboxes yields the association between
function calls and functions, enabling a graph that represents the behavior of malware
to be constructed. Using the application programming interfaces (APIs) that are called
by the software as nodes, the call relationships between APIs as edges, and the
underlying semantics of APIs as node features, the behavior of malware is obtained by
subgraph integration. The results show that the accuracy and precision of the detection
model are 0.945 and 0.95, respectively, and the accuracy and precision of the
classification model are 0.926 and 0.93, respectively. These results are better than those
for previously developed methods.

I
n recent years, following the development of informa-
tion technology, digital transformation has become
necessary for governments and major enterprises.
Information technology is indispensable in the develop-
ment of innovative products and services, establishing
new business models, or digitalizing the production
and supply chains of chemical plants. However, new
cyberthreats have invaded the Internet of Things (IoT)
domain from the traditional information environment
and become an omnipresent threat. According to the
Global Information Security Threat Report published by
Fortinet, the number and sophistication of attacks
against individuals, enterprise organizations, and critical
infrastructure globally increased significantly in the first
half of 2021.1 Therefore, enterprises of all sizes are
actively addressing cyberattacks in all instances of risk.
Enterprises need to be proactive in preventing damage
by attackers.
Owing to the impact of COVID-19, many cybercrimi-
nals have exploited COVID-19 to spread malicious files
1520-9202 © 2023 IEEE
Digital Object Identifier 10.1109/MITP.2023.3264509
Date of current version 30 June 2023.
and send phishing e-mails that contain COVID-19-
related information. Due to the severity of COVID-19,
the average user often does not pay attention to the
source of the message or the files that are attached
to the message, allowing attackers to gain access to
their victims’ systems.2 In addition, the digitalization of
factories and businesses is a major global trend, and
companies are introducing big data and artificial intelli-
gence to enhance their competitiveness. As a result,
many devices will transmit more data over the Internet,
including, of course, commercially valuable and confi-
dential information. These devices and information will
become the target of attackers. Many attacks in recent
years have been launched by state-level organizations,
often on critical national facilities. According to antivi-
rus test (AV-TEST) statistics that were published in
2022, the number of types of malware that target Win-
dows operating systems has grown steadily since 2018
at 10–25% annually.3
Most current strategies of defense against malware
are based on static analysis methods, such as signa-
ture matching. The advantage of static analysis is that
it covers the entire code of a piece of malware and can
be conducted without executing it. However, some
defense strategies that are based on dynamic analysis

Authorized
May/June licensed use limited to: Northeastern
2023 PublishedUniversity. Downloaded
by the IEEE Computeron July 28,2023 at 12:02:25 UTC from
Society IT IEEE Xplore. Restrictions apply.
Professional 43
IT SECURITY-VULNERABILITY DETECTION
have also been developed; these use such features as
sensitive behavior, critical access to systems, network
traffic analysis, and key program monitoring.4 Since
dynamic analysis is performed on the actual behavior
of malware, it can effectively detect and classify mal-
ware variants. Most of the approaches focus on specific
types of malware, so developing new and generalized
detection methods is essential.
This study proposes an efficient way to analyze the
behavior of a Windows portable executable (PE). Graph
convolutional networks (GCNs) and function call graphs
are used to detect and classify malware. This study will
identify and classify software types using neural net-
works and sandbox analysis results. Researchers will
be able to use this method to classify software types
quickly.
The contributions of this article to its field are as
follows:
An approach to classify software types that is
based on dynamic analysis is developed. Malware
is run in a sandbox, and the function call relation-
ships are analyzed to generate call graphs.
The Weisfeiler–Lehman graph kernel-based GCN
(WL-GCN) model is used for general malware
detection. A malware multiclassification task is
carried out so that the samples can be identified
as malware by model analysis.
The accuracy that is achieved in this work is
around 7% higher than that achieved in previ-
ous studies. By enhancing the node features,
the call graph can be made to better represent
the behavior and patterns of malware.
RELATED WORKS
Malware is an all-embracing term. Any software that
acts maliciously or is intended to damage and exploit a
programmable device can be called malware. Cyberat-
tackers often use malware to obtain sensitive informa-
tion with the aim of gaining financial benefit at the
expense of the victim. Stolen data are usually credit
card information and personal account passwords. In
recent years, due to the popularity of cryptocurrencies,
attackers have also planted malware on victims’ devi-
ces to “mine” them for cryptocurrency. Each type of
malware has different objectives, such as stealing infor-
mation from victims’ accounts, running forced adver-
tisements, and spamming and extorting money. This
chapter introduces recent malware-related research,
including signature-based techniques, heuristic techni-
ques, and recent work on machine learning and deep
learning techniques.
44 Authorized licensed
ITuse limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from IEEE Xplore. Restrictions
Professional
Signature-Based Mechanism
Much antivirus software relies on file signatures to
detect malware. This technique involves reading or scan-
ning a file to determine whether it matches a signature
in the signature database. The signature information
includes the file size, the function that is used in the
sample, data bytes at certain positions, the printable
string, and the hash value of the entire file. Signatures
can be generated automatically using appropriate tools.4
The advantage of this method is that it can detect mal-
ware quickly, is simple and lightweight, and has a low
false-positive rate. However, it is susceptible to generat-
ing false positives when new software is encountered,
whether the sample is malicious or benign. Since these
new software signatures do not appear in the database,
analysts must maintain the signature database regularly
to ensure the accurate detection of malicious software
and reduce the false-positive rate.
Naik et al.5 proposed a fuzzy–import hashing tech-
nique, which integrates two methods: fuzzy hashing
and import hashing. The general hashing algorithm
identifies whether files are identical. The principle of
fuzzy hashing is to divide a file into several blocks to
perform hashing operations separately on them; the
similarity is then calculated using this hash value.
Many factors affect fuzzy hashing, including the file
size, block size, and type of hash function. Associated
methods include SSDEEP,6 SDHASH,7 and mvHASH-B.8
The study compensates for the shortcomings of both
fuzzy hashing and import hashing by integrating them.
The results thus obtained show that this integrated
approach improves the rate of analysis and the detec-
tion of malware and is much more effective than fuzzy
hashing or import hashing alone. However, it is based
on the structural or syntactic similarity or does not
detect behavioral and semantic similarities, disfavoring
the detection of new malware.
Heuristic-Based Mechanism
Signature-based detection techniques are not effective
in detecting unknown malware, but heuristic-based
detection techniques can infer sample behavior through
a series of features to identify unknown malware. Heu-
ristic detection techniques can be divided into static
and dynamic depending on the types of features used.9
Static heuristics detection enables analysis without
executing suspicious files, using, for example, printable
strings, OpCode, and control flow diagrams—features
that are normally available through decompilation
techniques. These features can be compared to mal-
ware features to recover the behavior of the program
and to detect new types of malicious samples. The
May/June apply.
2023

advantages of this approach are the lack of need to
run malware samples during the analysis as well as the
speed and low cost. However, malware developers can
make static heuristics detection less efficient though
code obfuscation and the use of packer techniques.10
Dynamic heuristics detection is also often referred
to as a behavior-based approach, wherein an isolated
system environment actually simulates and records
malware behavior and determines whether a sample is
malicious or benign. This type of approach usually fea-
tures API call, file system, and registry operations to
record the actual interaction of malware with the
system and to infer whether its behavior is malicious
or not.
Pektaş and Acarman11 used mining and searching
n-gram algorithms on API sequences to represent
the behavioral features of malware, and they thereby
extracted malicious API patterns using a proposed vot-
ing algorithm to identify the malware type. Cabau
et al.12 proposed an automatic classification system
that classifies suspicious samples as malicious or
benign based on the characteristics of the file system,
registry operations, and network access operations.
Their study was the first to extract behavioral features
from an isolated environment; quantify those features
using a proposed algorithm; and, finally, use the sup-
port vector machine classifier for identification.
Dynamic heuristic detection is less affected than
static heuristic detection by code obfuscation and
packer technologies. However, dynamic heuristic detec-
tion also has shortcomings. First, the isolated environ-
ment and the associated dynamic analysis tools are
easily detected by malware, which will then evade those
tools, resulting in poor analysis efficiency. Another dis-
advantage is that the method is costlier and slower
than static heuristic detection. Therefore, both methods
can be combined to compensate for the shortcomings
of each and, thus, effectively deal with inverse analysis
technologies. For example, the use of antidebugging or
code obfuscation techniques by malware developers
seriously affects the results of static analysis, but not
those of dynamic analysis. Antidebugging techniques
that target more than two types of analysis are not stan-
dard in common malware.10
Artificial Intelligence-Based
Mechanisms
Machine learning requires a manual selection of the
features of data, which are then learned and classified
using a model. The selected features significantly
affect the classification results of the model; deep
learning can support the automatic learning of features
Authorized
May/June licensed use limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from
2023
IT SECURITY-VULNERABILITY DETECTION
from data by simulating the learning and operation pro-
cess of human neurons. Although the computational
and resource costs are higher than those of machine
learning, the predictions are usually better.
Sharma et al.13 proposed a detection method that is
based on OpCode execution frequency; it uses the
Fisher score, information gain, and chi-square for feature
selection and uses the random forest, logistic model
tree, and Named Binary Tags (NBT) classifier in Waikato
Environment for Knowledge Analysis (WEKA) for mal-
ware detection. Their results show that this method can
effectively detect malware. Nguyen et al.14 proposed an
IoT botnet detection approach that is based on a print-
able string information graph and deep graph convolu-
tional neural network classifier. They used the IDA Pro
tool to generate control flow graphs, printable string
information to build call graphs, and convolutional neu-
ral networks (CNNs) for classification.
Yuan et al.15 proposed a byte-level malware classifi-
cation method that is based on Markov images and
deep learning. The results thus obtained showed that
this method is more efficient than gray-scale image-
based methods. Various authors16–18 have performed
the image transformation of executable files and used
CNN models (e.g., VGG16 and ResNet-50) to classify
malware.
Li et al.19 proposed a classifier that is based on a
GCN. Their method uses malware API sequences to
generate a directed cyclic graph. The results show that
the method is effective in detecting malware and out-
performs previously studied methods. Hung et al.20
proposed a malware detection method that is based
on a multiedge dataflow graph representation and a
CNN. They also proposed a model framework called
MalGCN for malware detection.
PROPOSED METHODS
This section introduces an overview of the proposed
methods and describes the dataset and model in detail.
Figure 1 shows the proposed system, which has four
main parts—data collection, dataset creation, detec-
tion model and prediction, and classification model and
prediction.
Data Collection
HatchingTriage (https://tria.ge/) is an online sandbox ser-
vice that is provided by Hatching International B.V. in
The Netherlands; it can perform dynamic software analy-
sis on various platforms. HatchingTriage stores many
malware samples, providing the family name, tags, and
Secure Hash Algorithm 256 (SHA256) hash of each mal-
ware sample. PortableFreeware (www.portablefreeware.
IT IEEE Xplore. Restrictions apply.
Professional 45
IT SECURITY-VULNERABILITY DETECTION
FIGURE 1. System overview. API: application programming interface; PE: portable executable.
com) provides many samples of benign software. This
study crawls these samples from PortableFreeware
using Python crawler technology. The sample files that
are collected in this study are of the Windows PE type.
One thousand samples of each of eight types of malware
were downloaded from Triage. A total of 8000 malicious
samples and 2000 benign samples were thus collected.
The six categories of malicious samples are adware,
backdoor, downloader, dropper, miner, and ransom.
Data Preprocessing
Figure 2 presents the steps of dataset creation. The
first step is to submit the samples to the Cuckoo Sand-
box for dynamic analysis and, thus, to obtain the exe-
cution analysis report and API execution sequence.
The second step is to draw the call graph based on the
API execution sequence. Then, the API names in the
API execution sequence are used to train the word2vec
model to convert the API information into a vector,
which is used as the feature of each node of the graph.
Sandbox Analysis
To collect information about the behavior of malware
during execution, Cuckoo Sandbox is used to perform
a dynamic analysis. The sandbox provides a secure
execution environment to prevent malware from infect-
ing an entire system. After the samples are run, the
FIGURE 2. Dataset creation steps.
46 Authorized licensed
ITuse limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from IEEE Xplore. Restrictions
Professional
sandbox provides information about the file system,
registry values, network traffic, and function calls; it
generates analysis reports in JavaScript Object Nota-
tion format, as shown in Figure 3.
By tracking the execution of each sample with
Cuckoo Sandbox, the API call sequence of all pro-
cesses during the execution period are obtained, and
the respective call graph for each process is drawn, as
shown in Figure 4. After the API execution sequence is
received, the sequence is converted into a graph,
which is defined as follows: G ¼ fV , E g, where V is the
set of vertices, V ¼ fv1 , v2 , v3 , :::, vn g, E is set of edges,
and E ¼ fðx, yÞjðx, yÞ 2 V 2 g. Each vertex represents an
API function.
When API A is executed and then API B is executed,
an edge is created between nodes A and B. The direc-
tion of the edge depends on the order of execution of
the two APIs. During execution, two identical APIs may
be executed consecutively so that the graph that is
drawn in this way is a directed cyclic graph. The bene-
fits of employing a call graph include the following:
1) the quick comprehension of code, such as the ability
to locate subfunctions that are not called by other pro-
grams; 2) the ability to monitor the change of variable
values in each subfunction for subsequent analysis;
and 3) the identification of irregular program execution,
such as a code injection attack.
May/June apply.
2023

FIGURE 3. Sandbox analysis reports.
API Embedding
The call graph that is generated in these steps has cap-
tured only an association between APIs and APIs with-
out any of the characteristics of APIs, so this section
explains how to perform API embedding. This study is
based on the use of API pairs proposed by Shamsi
et al.,21 in which the upward and downward call relation-
ships of APIs are learned through Word2Vec and vec-
torized. The category of each API—such as file, system,
or resource—is added to vectors to represent their
behavioral features. This study is based on the Gensim-
Word2Vec22 open source library for API embedding. The
corpus of the model was built using API sequences
from sandbox analysis reports, and a total of 6000 soft-
ware samples were used to generate 301 API functions.
The training model uses a skip-gram to obtain the
semantics of the current API by the preceding and fol-
lowing APIs; it uses these word vectors as node fea-
tures of the call graph.
FIGURE 4. Call graph.
Authorized
May/June licensed use limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from
2023
IT SECURITY-VULNERABILITY DETECTION
Call Graph Integration
Many non-PE software samples were found during
sandbox analysis. The file command in Linux was used
to filter and classify the files. After sandboxing and the
generation of the call graph, it was found that, among
the dataset with a total of 10,000 samples (8000 mali-
cious and 2000 benign), more than 1000 samples con-
tained more than 200 nodes, and some of those
contained even more than 1000 nodes, as shown in
Figure 5, because the samples generated many behav-
iors during execution, such as creating other pro-
cesses, copying themselves, and operating through
other system programs, causing many API calls and
producing a large number of nodes in the call graph.
Therefore, in this study, the nodes in the call graph are
pruned by the following method:
[ [
G ¼ Sumðgi , gj Þ gi  Sumðgi , gj Þ fgj  Sumðgi , gj Þg
where gi ¼ ðNi , Ei Þ and gj ¼ ðNj , Ej Þ are the call graph
of the process generated during the execution of the
sample. By overlaying the subcall graph (gi and gj ), the
behavioral process of each process is added to G. The
“Sum” indicates that the convolution output is summed.
In this way, the behavioral information associated with
each process can be maintained while the number of
graph nodes is reduced. After pruning, the number of
call graph nodes is below 200.
“Trim” removes a redundant part of a model, and
“unTrim” means to not remove a redundant part of a
model. Figure 5 presents the statistics after pruning.
Some of the samples that were collected during the
data collection phase could not be executed directly in
the sandbox. Some of those samples are of types, such
as Microsoft Excel spreadsheet, Visual Basic Script,
and others, that are not included in this study; only the
executable type is considered herein.
FIGURE 5. Nodes in call graph.
IT IEEE Xplore. Restrictions apply.
Professional 47
IT SECURITY-VULNERABILITY DETECTION

FIGURE 6. Model architecture. GCN: graph convolutional network.

Model Architecture the LogSoftMax function and transformed into the

Figure 6 shows the architecture of the classification
model. First, GCNConv, which is an aggregate and
combined strategy for learning these graph nodes and
structural information, is used. Then, the GCN convolu-
tional layer is used for three iterations, and the features
of each node are stacked. Second, the global sort pool
and 1-D CNN, which is a readout strategy, are used to
sort features based on the last layer of the GCN; then,
the top 30 values are selected, and the 1-D CNN and
the full connection layer are entered to transform any
simple graph into an embedding vector. Third, the linear
transformation is applied to the embedding vector to
convert it into two dimensions (two dimensions for the
detection task and eight dimensions for the multiclassi-
fication task). Finally, the embedding vector is input to
probability distribution of each type of sample. Table 1
shows the parameters of the model. In this classifica-
tion model, the learning rate is 103 , the batch size is
100, the optimization function is Adam, and the loss
function is CrossEntropyLoss.
PERFORMANCE ANALYSIS
The following section describes the experimental envi-
ronment and compares the system performance with
the performance of the previously developed system.
Experimental Environment
Table 2 lists the hardware devices used in this study.
Hardware includes an AMD Ryzen 7 3700x eight-core
CPU at 3.6 GHz, a DDR4 2  32-GB global sort pool

TABLE 1. Parameters of the model.*

Layer (Input, Output) Activation Remarks

GCNConv (N, 32) Tanh Convolution
N is the features of the node.
GCNConv (32, 32) Tanh Convolution
GCNConv (32, 32) Tanh Convolution
GCNConv (32, 1) Tanh Convolution
Global sort pool K ¼ 30
Conv1d (1, 16) ReLu Convolution
MaxPool1d Kernel size ¼ 2,
stride ¼ 2
Conv1d (16, 32) ReLu Convolution
Linear (352, 128) ReLu
Linear (128, X) Log SoftMax X is the number of classes.
*ReLu: rectified linear unit; TanH: tangens hyperbolicus.

48 Authorized licensed
ITuse limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from IEEE Xplore. Restrictions
Professional May/June apply.
2023
 IT SECURITY-VULNERABILITY DETECTION

TABLE 2. Hardware specifications.* TABLE 4. Dataset distribution.

Item Specifications Trainable

Number Type Downloaded data
CPU AMD Ryzen 7 3700x eight-core CPU at
3.6 GHz 1 Adware 1000 223
RAM DDR4 2  32-GB RAM 2 Backdoor 1000 415
HDD 2-TB HDD 3 Downloader 1000 364
GPU NVIDIA GeForce GTX 1080 8 GB 4 Dropper 1000 757
*RAM: random-access memory; HDD: hard disk drive. 5 Miner 1000 784
6 Ransomware 1000 633
random-access memory, a 2-TB HDD, and an NVIDIA
7 Spyware 1000 368
GeForce GTX 1080 8-GB graphics card.
Since it is not certain whether the behavior of the 8 Worm 1000 187
suspect sample will cause system damage, dynamic anal- 9 Benign 2000 1784
ysis requires that the sample be executed in an isolated
environment to perform the sample’s dynamic behavior
analysis safely. This study uses the Ubuntu operating sys- Results of the Analysis
tem as a Cuckoo host and Windows 7 64 bit as a guest Detection Task
In this study, 150 epochs were used for the detection
system. The guest system is installed on VirtualBox, and
part of the training. This number was chosen because
Microsoft Office, Adobe Acrobat Reader DC, and other
the model would be overfitted after more than 150
common user applications are installed beforehand to
epochs. Figure 7 shows the training history of the
match real-world usage conditions closely. Table 3 pro-
model for the detection task. As seen in the figure, the
vides host and guest system information.
model improves and converges quickly after about 10
epochs. The unbalance of the dataset causes the
Dataset Distribution model to prefer classes with a higher number of predic-
Table 4 shows the distribution of data in the datasets. tions in the initial training phase. After 150 epochs, the
The number of downloads, type name, and data that training and validation accuracies of the model were
finally were used in training are shown for each sample around 0.97.
type. The dataset contains adware, backdoor, down- After the model was trained, a test dataset was
loader, dropper, miner, ransomware, spyware, worm, used to make predictions. The predictions were con-
and benign, for a total of 3731 malicious samples and verted into a confusion matrix. According to the confu-
1784 benign samples. sion matrix, the predictions of the model are biased
To allow the classification model to be trained and toward malware. The imbalance in the dataset causes
validated correctly in each epoch, the test data that the predictions of the model to be biased because there
are not included in the training set are used for reason- is about twice as much malware as benign software.
able model performance evaluation. In this study, data However, the prediction accuracy for benign software is
concerning 100 randomly selected samples of each still 89%. Figure 8 shows the indicators. The predictions
sample type in the dataset were used as test sets, and of the test dataset had an accuracy of 0.945, a precision
the remaining data were divided into 90% for the train- of 0.95, a recall of 0.945, an F1 score of 0.945, and a
ing dataset and 10% for the validation dataset. macro area under the curve (AUC) of 0.94.
TABLE 3. Host and guest system information.*
Multiclassification Task
Item Specifications The model was trained for 200 epochs in the multiclassi-
fication task to prevent overfitting, as in the malware
Cuckoo Sandbox Version 2.0.7
classification tasks. Figure 9 shows the training history
Host OS Ubuntu 16.04 LTS of the model in the multiclassification task. It shows that
Host RAM 16 GB the model improves and converges rapidly after about 10
epochs. After 200 epochs, the training accuracy of the
Guest OS Windows 7 64 bit
model is 96%, and the validation accuracy is 90%.
Guest RAM 4 GB After the model has been trained, a test dataset is
*OS: operating system. used to make predictions. The predictions are plotted

Authorized
May/June licensed use limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from
2023 IT IEEE Xplore. Restrictions apply.
Professional 49
IT SECURITY-VULNERABILITY DETECTION

FIGURE 7. Training history for the detection task. AUC: area under the curve; ROC: receiver operating characteristic.

FIGURE 8. Performance for the detection task. acc: accuracy.

FIGURE 9. Training history for the multiclassification task.

50 Authorized licensed
ITuse limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from IEEE Xplore. Restrictions
Professional May/June apply.
2023
 IT SECURITY-VULNERABILITY DETECTION

FIGURE 10. Performance for the multiclassification task.

as a confusion matrix. After training with benign software,
the accuracy of the detection of adware and backdoor
improves because some of the behavioral characteristics
of these samples are similar to those of the detection
of benign software. However, the accuracy of ransom-
ware and benign is relatively low because some of the
ransomware samples use antisandboxing techniques
and do not perform any malicious operations after
entering the sandbox, making them similar to benign
samples. Figure 10 shows the performance metrics of
the model in the multiclassification task, including the
confusion matrix, the receiver operating characteristic
curve, and other metrics. The predictions of the test
dataset had an accuracy of 0.926, a precision of 0.930, a
recall of 0.926, an F1 score of 0.926, and a macro AUC
of 0.96.
Performance Comparison
This section measures the performance by comparing
the similarity model with previous studies. This study
uses the test dataset for comparison to ensure consis-
tency of the benchmark.
Li et al.19 proposed a classifier that was based on a
GCN. The method uses malware API sequences to gen-
erate a directed cyclic graph. However, they did not pro-
vide the details of their experiments. In particular, they
failed to explain 1) how the APIs were embedded,
2) how the call graph of each program was integrated
into the dynamic analysis, and 3) the model hyperpara-
meters. Therefore, the approach attempted to be imple-
mented in this study differs from that of Li et al. but is
conceptually similar to it. For the GCN model to per-
form learning and graph classification tasks on a call
graph to represent the behavior of the whole graph,
nodes must be defined in advance. In addition, the
nodes of various graphs differ. This study uses one-hot
encoding for API embedding, potentially generating too
many nodes, but the method suffices to represent the
behavior of and information about each API. Global_
Max_Pool is the readout strategy and, finally, SoftMax
is used to output the final result.
Table 5 compares the performance of the graph
method and model that are developed in this study
to that achieved in other studies. The graph method of

TABLE 5. Performance metrics of comparison with previous studies.*

Detection task Multiclassification task

Graph method Markov chain Proposed method Markov chain Proposed method
Model GCN WL-GCN WL-GCN WL-GCN
Precision 0.896 0.950 0.856 0.930
Recall 0.875 0.945 0.856 0.926
F1 score 0.873 0.945 0.854 0.926
Accuracy 0.875 0.945 0.856 0.926
*GCN: graph convolutional network; WL-GCN: Weisfeiler–Lehman graph kernel-based graph convolutional network.

Authorized
May/June licensed use limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from
2023 IT IEEE Xplore. Restrictions apply.
Professional 51
IT SECURITY-VULNERABILITY DETECTION
Li et al.19 uses Markov chain-based methods to gener-
ate call graphs and perform graph classification. In the
model part, they use a traditional GCN to detect mal-
ware. As the model that was used by Li et al. did not
perform well in the multiclassification task, the graph
method is compared with the Markov chain-based
approach only in the multiclassification task.
CONCLUSION
This study proposes and successfully applies a method
for malware detection and classification that is based
on GCNs. Samples of software are analyzed using a
sandbox to obtain the actual behaviors and functions
that are executed by the malware and to build a func-
tion call graph. Then, the behaviors and features of each
API are represented by embedding the API call sequen-
ces into the call graph. Finally, the API representation
vector is used as a feature in the call graph nodes, and
the function call graph can represent the malware
behavior. GCNs are used for malware detection and
classification. Our future work will focus on antidetec-
tion, including antisandboxing technology; doing so will
make the analysis more complicated. Future research
on antidetection must be conducted to reduce malware
evasion. In addition, by mining the common APIs of mal-
ware and adding to the model, such as the graph atten-
tion network model, the call graph can become more
discriminating.
REFERENCES
1. “Global threat landscape report,” Fortinet, Sunnyvale,
CA, USA, Aug. 2021. Accessed: Apr. 20, 2022. [Online].
Available: https://www.fortinet.com/content/dam/
maindam/PUBLIC/02_MARKETING/08_Report/report-
2021-threat%20landscape.pdf
2. “Sonicwall cyber threat report,” SonicWall, Milpitas,
CA, USA, 2021. Accessed: Apr. 21, 2022. [Online].
Available: https://www.sonicwall.com/medialibrary/
en/white-paper/2021-cyber-threat-report.pdf
3. “AV-ATLAS analyzes.” AV-TEST. Accessed: Mar. 5,
2022. [Online]. Available: https://portal.av-atlas.org/
4. N. Naik, P. Jenkins, R. Cooke, J. Gillett, and Y. Jin,
“Evaluating automatically generated YARA rules and
enhancing their effectiveness,” in Proc. IEEE Symp.
Ser. Comput. Intell., 2020, pp. 1146–1153, doi: 10.1109/
SSCI47803.2020.9308179.
5. N. Naik, P. Jenkins, N. Savage, L. Yang, T. Boongoen,
and N. Iam-On, “Fuzzy-import hashing: A static
analysis technique for malware detection,” Forensic
Sci. Int., Digit. Investigation, vol. 37, Jun. 2021,
Art. no. 301139, doi: 10.1016/j.fsidi.2021.301139.
52 Authorized licensed
ITuse limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from IEEE Xplore. Restrictions
Professional
6. J. Kornblum, “Identifying almost identical files using
context triggered piecewise hashing,” Digit.
Investigation, vol. 3, pp. 91–97, Sep. 2006, doi: 10.1016/
j.diin.2006.06.015.
7. V. Roussev, “Data fingerprinting with similarity
digests,” in Proc. Adv. Digit. Forensics VI, 2010,
pp. 207–226, doi: 10.1007/978-3-642-15506-2_15.
8. F. Breitinger, K. P. Astebøl, H. Baier, and C. Busch,
“mvHash-B – A new approach for similarity preserving
hashing,” in Proc. Int. Conf. IT Secur. Incident Manage.
IT Forensics, 2013, pp. 33–44, doi: 10.1109/IMF.2013.18.
€ A. Aslan and R. Samet, “A comprehensive review
9. O.
on malware detection approaches,” IEEE Access,
vol. 8, pp. 6249–6271, Jan. 2020, doi: 10.1109/ACCESS.
2019.2963724.
10. Z. Guo, W. Zhang, W. Yang, X. Che, Z. Zhang, and
M. Li, “A survey on feature extraction methods of
heuristic backdoor detection,” in Proc. Int. Conf.
Frontiers Electron., Inf. Comput. Technol., 2021,
pp. 1–7, doi: 10.1145/3474198.3478137.
11. A. Pektaş and T. Acarman, “Malware classification
based on API calls and behaviour analysis,” IET Inf.
Secur., vol. 12, no. 2, pp. 107–117, Mar. 2018,
doi: 10.1049/iet-ifs.2017.0430.
12. G. Cabau, M. Buhu, and C. P. Oprisa, “Malware
classification based on dynamic behavior,” in Proc.
18th Int. Symp. Symbolic Numer. Algorithms Scientific
Comput. (SYNASC), 2016, pp. 315–318, doi: 10.1109/
SYNASC.2016.057.
13. S. Sharma, C. R. Krishna, and S. K. Sahay, “Detection
of advanced malware by machine learning
techniques,” in Proc. Adv. Intell. Syst. Comput., 2019,
pp. 333–342, doi: 10.1007/978-981-13-0589-4_31.
14. H. T. Nguyen, Q. D. Ngo, and V. H. Le, “IoT botnet
detection approach based on PSI graph and DGCNN
classifier,” in Proc. IEEE Int. Conf. Inf. Commun. Signal
Process. (ICICSP), 2018, pp. 118–122, doi: 10.1109/
ICICSP.2018.8549713.
15. B. Yuan, J. Wang, D. Liu, W. Guo, P. Wu, and X. Bao,
“Byte-level malware classification based on markov
images and deep learning,” Comput. Secur., vol. 92,
May 2020, Art. no. 101740, doi: 10.1016/j.cose.2020.
101740.
16. F. Zhong, Z. Chen, M. Xu, G. Zhang, D. Yu, and
X. Cheng, “Malware-on-the-brain: Illuminating malware
byte codes with images for malware classification,”
IEEE Trans. Comput., vol. 72, no. 2, pp. 438–451,
Feb. 2022, doi: 10.1109/TC.2022.3160357.
17. D. Gibert, C. Mateu, J. Planes, and R. Vicens, “Using
convolutional neural networks for classification of
malware represented as images,” J. Comput. Virol.
Hacking Techn., vol. 15, no. 1, pp. 15–28, Mar. 2019,
doi: 10.1007/s11416-018-0323-0.
May/June apply.
2023

18. D. Vasan, M. Alazab, S. Wassan, B. Safaei, and
Q. Zheng, “Image-based malware classification using
ensemble of CNN architectures (IMCEC),” Comput.
Secur., vol. 92, May 2020, Art. no. 101748, doi: 10.1016/j.
cose.2020.101748.
19. S. Li, Q. Zhou, R. Zhou, and Q. Lv, “Intelligent malware
detection based on graph convolutional network,”
J. Supercomputing, vol. 78, no. 3, pp. 4182–4198,
Feb. 2022, doi: 10.1007/s11227-021-04020-y.
20. N. V. Hung, P. Ngoc Dung, T. N. Ngoc, V. Dinh Phai,
and Q. Shi, “Malware detection based on directed
multi-edge dataflow graph representation and
convolutional neural network,” in Proc. 11th Int. Conf.
Knowl. Syst. Eng. (KSE), 2019, pp. 1–5, doi: 10.1109/KSE.
2019.8919284.
21. F. Al Shamsi, W. L. Woon, and Z. Aung, “Discovering
similarities in malware behaviors by clustering of API
call sequences,” in Proc. Int. Conf. Neural Inf. Process.,
2018, pp. 122–133, doi: 10.1007/978-3-030-04212-7_11.

22. R. Rehůřek and P. Sojka, “Software framework for topic
modelling with large corpora,” in Proc. LREC Workshop
New Challenges NLP Frameworks, 2010, pp. 45–50.
HSIANG-YU CHUANG is a graduated student with the
Department of Electrical Engineering, National Taiwan
Authorized
May/June licensed use limited to: Northeastern University. Downloaded on July 28,2023 at 12:02:25 UTC from
2023
IT SECURITY-VULNERABILITY DETECTION
University of Science and Technology, Taipei, 106335, Taiwan.
His research interests include machine learning, deep learn-
ing, and malware detection. Chuang received his M.S. degree
in electrical engineering from National Taiwan University of
Science and Technology. Contact him at m10907504@gapps.
ntust.edu.tw.
JIANN-LIANG CHEN is a distinguished professor and dean of
the Department of Electrical Engineering, National Taiwan
University of Science and Technology, Taipei, 106335, Taiwan.
His research interests include cellular mobility management,
cybersecurity, personal communication systems, and the
Internet of Things. Chen received his Ph.D. degree in electrical
engineering from National Taiwan University. He is a Senior
Member of IEEE. Contact him at lchen@mail.ntust.edu.tw.
YI-WEI MA is an assistant professor with the Department of
Electrical Engineering, National Taiwan University of Science
and Technology, Taipei, 106335, Taiwan. His research interests
include the Internet of Things, cloud computing, and future
network. Ma received his Ph.D. degree in engineering science
from National Cheng Kung University. He is the corresponding
author of this article. Contact him at ywma@mail.ntust.edu.tw.
IT IEEE Xplore. Restrictions apply.
Professional 53