Learning Deep Boltzmann Machines using Multi Ant

Colony Optimization to discover intrusion, anomaly

and cyber attacks

Asalah A. Al-Swidi Prof. Basher M. Al-Magalh, PhD

Thamar University Thamar University
Master's Student in Computer Science Dean of the computing college
Thamar, Yemen Thamar, Yemen

I. INTRODUCTION
In the last few years, the field of artificial intelligence has
developed greatly, especially in the field of deep learning, where
hardly a day goes without hear the latest advancements and
improvements that artificial intelligence and deep learning have
brought to a wide spectrum of domains from medicine,
engineering, science and many others. Deep learning has
become a science in itself at the present time, which is inspired
from the mechanism of the brain's work, its beginning was the
emergence of neural networks, as a result of their development,
the increase in the number of nodes and the number of layers in
them, deep learning appeared. Deep learning networks have
developed greatly in recent times and have been used in various Fig.1 Deep Boltzmann Machine
fields, as several algorithms have appeared like
backpropagation, autoencoders, variational autoencoders, This algorithm is usually trained using an algorithms based
restricted Boltzmann machines, deep Boltzmann machine, deep on probabilities and derivations like algorithms based on
belief networks, convolutional neural networks, recurrent neural Markov Chain Monte Carlo (MCMC), these algorithms like:
networks, generative adversarial networks, capsnets,
transformer, embeddings from language models, bidirectional • Variational Learning
encoder representations from transformers, and attention in • Stochastic Approximation Algorithm
natural language processing [1]. Each of these algorithms has its
own working mechanism and has its uses, applications, • Adaptive MCMC
advantages, disadvantages and challenges, as it is classified
either supervised, unsupervised or semi-supervised algorithms. For more details about these algorithms refer to [2]. Where
MCMC algorithms use these equations:
In this proposal, the deep Boltzmann machine was used,
which is a development of the restricted Boltzmann machine The energy of the state {v, h} is defined as:
(RBM), RBM is an unsupervised learning algorithm that can 𝐸(𝑣, ℎ; 𝜃) = −𝑣 𝑇 𝑊 1 ℎ1 − ℎ1𝑇 𝑊 2 ℎ2 (1)
automatically find patterns in its data by reconstructing the 1 2
inputs, it is a network consisting of two layers the visible layer Where v is the visible layer, h = {h , h } are the set of hidden
and the hidden layer, it has no output layer and it has units, and θ = {W1, W2} are the model parameters, representing
symmetrical bipartite graph because every node in the visual visible to hidden and hidden to hidden symmetric interaction
layer is connected to every node in the hidden layer and every terms. The probability that the model assigns to a visible vector
node in the hidden layer is connected to every node in the visible v is:
layer, it works both front and back and this work is the most 𝑃∗ (𝑣; 𝜃) 1
important to discover patterns and features, in which weights 𝑃(𝑣; 𝜃) = = ∑ exp(−𝐸(𝑣, ℎ1 , ℎ2 ; 𝜃)) .
and biases play an important role, it is called constrained because 𝑍(𝜃) 𝑍(𝜃)
ℎ
there is no communication between nodes of the same layer, The derivative of the log-likelihood with respect to
DBM is RBM but with many of the hidden layers that help to parameter vector W1 takes the following form:
discover patterns and features accurately and efficiently as in
Figure 1.
 𝜕 log 𝑃(𝑣; 𝜃) 𝑇 𝑇
1
= 𝐸𝑃𝑑𝑎𝑡𝑎 [𝑣ℎ1 ] − 𝐸𝑃𝑚𝑜𝑑𝑒𝑙 [𝑣ℎ1 ], (2)
𝜕𝑊
Where EPdata [·] denotes an expectation with respect to the
completed data distribution Pdata (h, v; θ) = P (h|v; θ) Pdata(v),
1 approach for face modeling is introduced. This paper presents a
with Pdata(v) = ∑𝑛 𝛿(𝑣 − 𝑣𝑛) representing the empirical
𝑁 novel Deep Appearance Models (DAMs) approach, an efficient
distribution, and EPmodel [·] is an expectation with respect to the
distribution defined by the model. The derivatives with respect
to parameters W2 take similar form.Exact maximum likelihood
learning in this model is intractable, because both data-
dependent and model-dependent expectations cannot be
computed analytically in less than exponential time.[3]The most
disadvantages of these algorithms. It is slow, difficult to train
and adjust weights, and unable to track the loss that is required
(let alone take derivatives with respect to it) where DBM is
significantly slower than backpropagation. The deep Boltzmann
machine algorithm is used in several applications, including
dimensionality reduction, classification, regression,
collaborative filtering, property learning, and modeling [1] and
it was used to extract more complex or sophisticated features and
hence can be used for more complex tasks. It was used in the
medical field, image analysis, feature discovery, cybersecurity,
which is what the deep Boltzmann machine will be trained for it
in this research, there are many researches that used DBM in
cybersecurity such as intrusion detection, attacks, anomaly in
networks and others, in all these researches the trained DBM
with an algorithm based on probabilities and derivations was
very precision compared to the rest algorithms but no study has
trained DBM using multi ant colony algorithm for any of the
researched areas.
The multi ant colony optimization algorithm will use to train
the deep Boltzmann machine in this research, this algorithm is a
development of the ant colony optimization algorithm, which is
one of the algorithms inspired by nature, it is inspired by the
swarms of ants, it was used to determine the best path used at
the beginning randomly and then the path is optimized by the
pheromone, on which the optimization process depends on it for
the rest of the ants. It is a method that depends on the pheromone
and the probabilities to choose the shortest path. This algorithm
has been developed, where the start is made with several ants
and the improvement is done in parallel, and thus it is faster than
the traditional algorithm with the number of ants and it is more
efficient.
II. RELATED WORKS
In [1], A Review of Deep Learning Algorithms and Their
Applications in Healthcare is proposed. This review aims to
categorically cover several widely used deep learning
algorithms along with their architectures, their practical
applications, challenges, The pros and cons, their applications in
healthcare and the future direction of this domain.
In [4], Deep medical image reconstruction with
autoencoders using deep Boltzmann machine training is
suggested. In this method Deep Autoencoder is implemented as
it has been considered for high dimensionality reduction. Layer
by layer pretraining is achieved using an approximate inference
algorithm called Deep Boltzmann Machine, the proposed
method proves to be efficient when compared with the
performance of the other autoencoders such as Deep
Autoencoder with multiple Backpropagation (DA-MBP), Deep
Autoencoder with RBM (DA-RBM) and Deep Convolutional
Autoencoder with RBM (DCA-RBM).
In [5], Deep appearance models: A deep Boltzmann machine
replacement for AAMs, to accurately capture both shape and
texture of face images under large variations. In this approach,
three crucial components represented in hierarchical layers are
modeled using the Deep Boltzmann Machines (DBM) to
robustly capture the variations of facial shapes and appearances.
Comparing to AAMs and other deep learning-based approaches,
the proposed DAMs achieve competitive results in those
applications, thus this showed their advantages in handling
occlusions, facial representation, and reconstruction.
In [6], Listening with your eyes: Towards a practical visual
speech recognition system using deep Boltzmann machines is
suggested. This method presents a novel feature learning method
for visual speech recognition using Deep Boltzmann Machines
(DBM). Unlike all existing visual feature extraction techniques
which solely extract features from video sequences, thus method
is able to explore both acoustic information and visual
information to learn a better visual feature representation.
In [7], Personalized prediction of genes with tumor-causing
somatic mutations based on multimodal deep Boltzmann
machine is proposed. An early cancer monitoring method is
developed based on multi-modal deep Boltzmann machine to
learn association between matched germline and somatic
mutations captured by whole exome sequencing from available
samples of cancer patients, and predict patient-specific high-risk
genes whose somatic mutations are required to drive normal
tissues to a tumor state. Our experiments on a set of breast cancer
samples show that our method significantly outperforms the
currently used frequency-based method in the personalized
prediction of genes carrying critical mutations.
In [8], Deep learning for cyber security intrusion detection:
Approaches, datasets, and comparative study is presented. In
this paper, a survey of deep learning approaches is present for
cyber security intrusion detection, the datasets used, and a
comparative study. Specifically, a review of intrusion detection
systems is provided based on deep learning approaches.
In [9], Deep learning techniques for cyber security intrusion
detection: A detailed analysis is suggested. In this study, a
detailed analysis of deep learning techniques is presented for
intrusion detection. Specifically, seven deep learning models is
analyzed, the most important performance indicators are used,
namely, accuracy, detection rate, and false alarm rate for
evaluating the efficiency of several methods.
In [10], A review of intrusion detection systems using
machine and deep learning in internet of things: Challenges,
solutions and future directions is proposed. This study aims to
present a comprehensive review of IoT systems-related
technologies, protocols, architecture and threats emerging from
compromised IoT devices along with providing an overview of
intrusion detection models. This work also covers the analysis
of various machine learning and deep learning-based techniques
suitable to detect IoT systems related to cyber-attacks.
 In [11], Application of deep learning architectures for cyber
security is presented. In this work, to leverage the application of
deep learning architectures towards cyber security, we consider
intrusion detection, traffic analysis and Android malware
detection. In all the experiments of intrusion detection, deep
learning architectures performed well in compared to classical
machine learning algorithms. Moreover, deep learning
architectures have achieved good performance in traffic analysis
and Android malware detection too.
In [12], Emergent deep learning for anomaly detection in
internet of everything is introduced. This paper introduces a new
generic deep learning frame- work for anomalous data detection
in internet of everything environments. It combines
decomposition methods, deep neural networks, and evolutionary
computation, a new recurrent neural network is proposed for
training time series data. Furthermore, two evolutionary
computation algorithms– the genetic and the bees swarm, are
proposed to accurately tuning the training step. These algorithms
consider the hyper-parameters of the trained models and attempt
to find the optimal values.
In [13], Improved Object Detection Algorithm using Ant
Colony Optimization and Deep Belief Networks Based Image
Segmentation is proposed. This paper represents that a variety
of strategies based on object detection and efficiency of object
detection framework using a saliency prior and DBNs for remote
sensing images. This research works proposed an efficient
object detection using the ant colony optimization and deep
belief networks.
In [14], A Novel Ant Colony Based DBN Framework to
Analyze the Drug Reviews is suggested, in this method a novel
Ant Colony- based Deep Belief Neural Network (AC-DBN)
framework is proposed. Drug review tweets are opted to perform
sentiment classification by using the proposed framework in
python environment. A model fitness function is initiated in the
DL framework and is observed that it is attaining high accuracy
with low computation time. Additionally, the obtained results
attained from the proposed framework are validated with
existing methods for evaluating the efficiency of the proposed
AC-DBN approach
In [15], Greedy learning of deep Boltzmann machine
(GDBM)’s variance and search algorithm for efficient image
retrieval is proposed. Initially, a preprocessing technique is
introduced. In this study, a technique that uses a median filter to
remove noise to achieve improved accuracy and reliability.
Then, Fourier and circularity descriptors are extract in an
effective manner correspondent to the texture and affine shape
adaptation features. In addition, various descriptors, such as
color histogram, color moment, color auto correlogram and
color coherency vector, are extracted as the invariant color
features. The multiple ant colony optimization (MACOBTC)
approach is implemented with whole features to find relevant
features. Finally, the relevant features are utilized for the greedy
learning of deep Boltzmann machine classifier (GDBM). The
proposed approach obtains effective performance and accurate
results on four datasets and is analyzed with various parameters
such as accuracy, precision, recall, Jaccard, Dice, and Kappa
coefficients. The GDBM provides a 25% increase in accuracy
compared with existing techniques, such as the a priori
classification algorithm.
III. PROBLEM STATMENT
Despite the successful applications of deep learning
algorithms, especially deep Boltzmann machine (DBM), which
has been used in many fields, but it is very difficult and slow in
the training process, it gets slower with the increase in the hidden
layers, and also one of its disadvantages is that it is unable to
track the loss that is required (let alone take derivatives with
respect to it). This is why this research was proposed, where the
deep Boltzmann machine algorithm will be trained using multi
ant colony optimization algorithm (MACO), due to what is
known about this algorithm in terms of speed and efficiency to
reach the best path and best solution in the shortest time.
Therefore, it will be used to reach the correct and required
weights and biases quickly to increase training speed and will be
used in the detection of cyber-attacks and detection of intrusion
anomalies in the networks that Its detection requires high speed
and extreme accuracy in order to respond to the threat in a timely
manner, as it is one of the time-critical applications.
IV. OBJECTIVES
The main objective of this research is to train Deep
Boltzmann Machine (DBM) with multi ant colony optimization
algorithm (MACO) to increase training speed and increase its
efficiency and effectiveness to reach the appropriate weights and
biases at high speed and precision
V. METHODOLOGY
In this proposed methodology, the algorithm will be built and
experiments will be conducted where will be input to DBM is
the behaviors that occur within networks that this input was
collected from datasets and preprocessing it first, and since it is
a symmetrical algorithm, its outputs are inputs to it, this process
continues until we reach the final output, which is attacks,
threats, intrusion and anomaly detection. Where security domain
was selected for the experiences because DBM was success in
discover intrusion, anomaly and cyber-attacks in previous
studies better than other algorithms. The DBM algorithm will
be trained with the MACO algorithm. What this algorithm needs
are four things, which are as follows:
• Ant: here are the inputs for DBM is the general
behavior of the network
• Pheromone: they are the weights and biases that the ant
controls on it and controls on its focus from increasing
or decreasing to reach the best and shortest path (the
best path is the best weights and biases that can reach
to trained DBM with the shortest way)
• Path: it is the beginning of the input (general behavior
within the network) to reach the output (Cyber-attacks
and detection of intrusions and anomalies in networks)
• Target: it is the shortest path, i.e., reaching the correct
weights and biases as quickly as possible, thus
speeding up the training of the network
 The algorithm will be applied in python environment
because it is accurate and easy to use with deep learning. After
the algorithm is implemented and applied, its accuracy and
effectiveness will be measured, its results will be compared with
previous studies and finally the conclusion and future works will
be presented.
VI. IMPORTANCE AND SCOPE
The importance of the research is to find a new way to train
DBM and increase the accuracy, efficiency and speed of its
results in the chosen field, which is cybersecurity and network
security field, as it was used to discover cyber-attacks, intrusions
and anomalies in networks that require high speed and accuracy
in order to respond in a timely manner.
VII. CONCLUSIONS
It was concluded that MACO succeeded in training DBM
with high speed and accuracy, in order to work in parallel and to
jointly improve weights and biases through the shared memory
among all ants, through the results presented by it compared to
previous studies, as this method was used in network security
and cybersecurity but now it can be generalized to any field in
which DBM can be used, as we recommend doing researches to
train it in other fields.
REFERENCES
[1] H. Abdel-jaber, D. Devassy, A. Al Salam, L. Hidaytallah and M. EL-
Amir, "A Review of deep learning algorithms and their applications in
healthcare," Algorithms . J.Kyoto. Switzerland, vol. 15, pp. 71, February
2022.
[2] R. Salakhutdinov and G. Hinton, "Deep Boltzmann machines," In
Proceedings of the International Conference on Artifical Intelligence and
Statistics, 2009, pp 448-455.
[3] R. Salakhutdinov, "Learning deep Boltzmann machine using adaptive
MCMC," In Proceedings of the 27th International Conference on Machine
Learning (ICML-10), 2010, pp. 943-950.
[4] S. Saravanan and S. Juliet, "Deep medical image reconstruction with
autoencoders using deep Boltzmann machine training," EAI Endorsed
Transactions on Pervasive Health and Technology . J . Belgium, vol. 6,
pp e2, September 2020.
[5] C. N. Duong, K. Luu, K. G. Quach and T. D. Bui, " Deep appearance
models: A deep boltzmann machine approach for face modeling," .
International Journal of Computer Vision. J. Springer. Netherlands, vol
127, pp. 437-455, December 2019.
[6] C. Sui and M. Bennamoun, " Listening with your eyes: Towards a
practical visual speech recognition system using deep boltzmann
machines," In Proceedings of the IEEE International Conference on
Computer Vision, 2015, pp. 154-162.
[7] Y. Li, F. Fauteux, J. Zou, A. Nantel and Y. Pan, " Personalized prediction
of genes with tumor-causing somatic mutations based on multimodal deep
Boltzmann machine," Neurocomputing. J. Elsevier. Netherland, vol 324,
pp. 51-62, February 2018.
[8] M. A. Ferrag, L. Maglaras, S. Moschoyiannis and H. Janicke, " Deep
learning for cyber security intrusion detection: Approaches, datasets, and
comparative study," Journal of Information Security and Applications. J.
Elsevier. United Kingdom, vol 50, pp. 102419, February 2020.
[9] M. A. Ferrag, L. Maglaras, H. Janicke and R. Smith, " Deep learning
techniques for cyber security intrusion detection: A detailed analysis," In
6th International Symposium for ICS & SCADA Cyber Security
Research, 2019, pp. 126-136.
[10] J. Asharf, N. Moustafa, H. Khurshid, E. Debie, W. Haider and A. Wahab,
" A review of intrusion detection systems using machine and deep
learning in internet of things: Challenges, solutions and future directions,"
Electronics. J. MDPI. Switzerland, vol 9, pp. 1177,July 2020.
[11] R. Vinayakumar, K.P. Soman, P. Poornachandran and S.
Akarsh,"Application of deep learning architectures for cyber security," In
Cybersecurity and Secure Information Systems, 2019, pp. 125-160.
[12] Y. Djenouri, D. Djenouri, A. Belhadi, G. Srivastava and J. Chun-Wei Lin,
" Emergent deep learning for anomaly detection in internet of
everything,", IEEE Internet of Things Journal. J. United States, 2021.
[13] A. Kaur and N. Kaur, "Improved Object Detection Algorithm using Ant
Colony Optimization and Deep Belief Networks Based Image
Segmentaion," International Journal of Latest Technology in Engineering,
Management & Applied Science IJLTEMAS. J. India, vol. 6, pp.86-90,
July 2017.
[14] N. Tazeen and K. S. Rani, " A Novel Ant Colony Based DBN Framework
to Analyze the Drug Reviews," International Journal of Intelligent
Systems and Applications. J. China, vol 13, pp. 25-39, December 2021.
[15] M. J. J. Ghrabat, G. Ma, Z. A. Abduljabbar, M. Al Sibahee and S. J.
Jassim, " Greedy learning of deep Boltzmann machine (GDBM)’s
variance and search algorithm for efficient image retrieval," IEEE Access.
J. United States, vol 7, pp.169142-169159,October 2019.