Send Orders for Reprints to reprints@benthamscience.

net
Current Medical Imaging, 2021, 17, 1-00 1
REVIEW ARTICLE

A Tour of Unsupervised Deep Learning for Medical Image Analysis

Khalid Raza1,* and Nripendra Kumar Singh2

1
Department of Computer Science, Jamia Millia Islamia, New Delhi, India; 2Department of Computer Science, Jamia
Millia Islamia, New Delhi, India

Abstract: Background: Interpretation of medical images for the diagnosis and treatment of com-
plex diseases from high-dimensional and heterogeneous data remains a key challenge in transform-
ing healthcare. In the last few years, both supervised and unsupervised deep learning achieved
promising results in the area of medical image analysis. Several reviews on supervised deep learn-
ing are published, but hardly any rigorous review on unsupervised deep learning for medical image
analysis is available.

ARTICLE HISTORY
Received: May 30, 2020
Revised: November 17, 2020
Accepted: December 16, 2020
DOI:
10.2174/1573405617666210127154257
Objectives: The objective of this review is to systematically present various unsupervised deep
learning models, tools, and benchmark datasets applied to medical image analysis. Some of the dis-
cussed models are autoencoders and their variants, restricted Boltzmann machines (RBM), deep be-
lief networks (DBN), deep Boltzmann machine (DBM), and generative adversarial network
(GAN). Future research opportunities and challenges of unsupervised deep learning techniques for
medical image analysis are also discussed.

Conclusion: Currently, interpretation of medical images for diagnostic purposes is usually per-
formed by human experts that may be replaced by computer-aided diagnosis due to advancement
in machine learning techniques, including deep learning, and the availability of cheap computing in-
frastructure through cloud computing. Both supervised and unsupervised machine learning ap-
proaches are widely applied in medical image analysis, each of them having certain pros and cons.
Since human supervisions are not always available or are inadequate or biased, therefore, unsuper-
vised learning algorithms give a big hope with lots of advantages for biomedical image analysis.

Keywords: Unsupervised learning, medical image analysis, autoencoders, restricted boltzmann machine, deep belief network.

1. INTRODUCTION supervised and unsupervised machine learning approaches

Medical imaging techniques, including magnetic reso-
nance imaging (MRI), positron emission tomography (PET),
computed tomography (CT), mammography, ultrasound, X-
ray, and digital pathology images, are frequently used diag-
nostic systems for the early detection, diagnosis, and treat-
ment of various complex diseases [1]. In the clinics, the im-
ages are mostly interpreted by human experts such as radiol-
ogists and physicians. Because of major variations in pathol-
ogy and the potential fatigue of human experts, scientists
and doctors have started using computer-assisted interven-
tions. The advancement in machine learning techniques, in-
cluding deep learning and the availability of computing in-
frastructure through cloud computing, has fuelled the field
of computer-assisted medical image analysis and comput-
er-assisted diagnosis (CAD). Deep learning is about learning
representations, i.e., learning intermediate concept or fea-
tures which are important to capture dependencies from in-
put variables to output variables in supervised learning, or
between subsets of variables in unsupervised learning. Both
*Address correspondence to this author at the aDepartment of Computer
Science, Jamia Millia Islamia, New Delhi, India; E-mail: kraza@jmi.ac.in
are widely applied in medical image analysis; each of them
has its pros and cons. Some of the widely used supervised
(deep) learning algorithms are Feedforward Neural Network
(FFNN), Recurrent Neural Network (RNN), Convolutional
Neural Network (CNN), Support Vector Machine (SVM),
and so on [2]. There are many scenarios where human super-
visions are unavailable, inadequate, or biased, and therefore,
a supervised learning algorithm cannot be directly used. Un-
supervised learning algorithms, including their deep architec-
ture, give hope with lots of advantages and have been wide-
ly applied in several areas of medical and engineering
problems, including medical image analysis.
This review presents unsupervised deep learning models,
their applications in medical image analysis, a list of soft-
ware tools/packages and benchmark datasets, and discusses
opportunities and future challenges in the area.
2. WHY UNSUPERVISED LEARNING?
In the majority of machine learning projects, the work-
flow is designed in a supervised way, where the algorithm is
humanly maneuvered, referring to tasks with already speci-
fied labels. In such supervised architecture, the potential of

1573-4056/21 $65.00+.00 © 2021 Bentham Science Publishers

2 Current Medical Imaging, 2021, Vol. 17, No. 00
the algorithms is limited in three ways: (i) A huge manual ef-
fort to create labels, (ii) To prepare biases to check the algo-
rithms’ supervised functioning, and (iii) Reduce the scalabili-
ty of the target function at hand.
To intelligently solve these issues, an unsupervised ma-
chine learning algorithm can be used. Unsupervised ma-
chine learning algorithms not only derive insights directly
from the data and group the data based on their patterns but
also use these insights for data-driven decision-making. Al-
so, unsupervised models are more robust in the sense that
they act as a base for several different complex tasks where
these can be utilized as the holy grail of learning and classifi-
cation. In fact, classification is not the only task that is em-
ployed, but other tasks such as compression, dimensionality
reduction, denoising, super-resolution, and some degree of
decision making are also performed. Therefore, it is more
useful to construct a model without having any knowledge
about the tasks. In a nutshell, we can think of unsupervised
learning as a preparation (pre-processing) step for super-
vised learning tasks, where the latter may allow better gener-
alization of a classifier [2].
Unsupervised pre-training gained popularity because of
its simplicity and has been highly addressed by the medical
imaging community. Moreover, autoencoders, along with
their several variants and other unsupervised methods, are re-
ported, which are being extensively applied in medical im-
age analysis.
3. UNSUPERVISED DEEP LEARNING MODELS
The unsupervised deep learning models can be broadly
classified as shown in Fig. (1). These have a myriad of uses
in different aspects of medical image processing.
3.1. Autoencoders and its Variants
3.1.1. Autoencoders and Stacked Autoencoder
Autoencoders (AEs) [3] are simple unsupervised learn-
ing models consisting of a single-layer neural network that
transforms the input into a latent or compressed representa-
tion by minimizing the reconstruction errors between input
and output values of the network. By constraining the dimen-
sion of latent representation (maybe from different input (s),
it is possible to discover relevant patterns from the data. Au-
toencoder framework defines a feature to extract function
with specific parameters [4]. Basically, AEs are trained with
a specific function. fθ is called encoder and h = fθ (x) is a fea-
ture vector or representation from input x. Another parame-
terized function gθ is called a decoder which produces input
space back from feature space. In short, basic AEs are
trained to minimize reconstruction error in finding a value of
parameter θ given by:
(1)
This minimization is optionally followed by a non-linear-
ity (most commonly used for encoder and decoder) as given
by:
Raza and Singh
(2)
(3)
where Sf and Sg are encoder and decoder activation func-
tion (normally sigmoid, hyperbolic tangent, or an identity
function), while the parameters of the model θ = {W, b, W',
d} where W and W’ are encoder-decoder weight matrices,
and b and d are encoder and decoder bias vector respective-
ly. Moreover, regularization or sparse constraints may be ap-
plied to boost the discovery process. In such a case, the hid-
den layer(s) has the same output as the input layer, and no
non-linearity is added. The model would simply learn an
identity function. (Fig. 2 and 3 illustrate the basic structure
of AE.
Fig. (1). Unsupervised deep learning models. (A higher resolution /
colour version of this figure is available in the electronic copy of
the article).
Stacked autoencoders (SAEs) are constructed by organiz-
ing AEs on top of one another, also known as deep AEs.
They consist of multiple AEs stacked into multiple layers
where the output of each layer is wired to the inputs of the
successive layers (Fig. 2(B)). To obtain good parameters,
SAE uses greedy layer-wise training. The benefit of this
type of autoencoder is that it can enjoy the benefits of a deep
network having greater expressive power. Furthermore, it
usually captures a useful hierarchical grouping of the input
[5].
3.1.2. Denoising autoencoder
Denoising autoencoder (DAE) is another variant of the
auto-encoder. Denoising is important and is encapsulated as
a training criterion for efficient and robust learning and to ex-
tract useful features [6]. They also prevent the model from
learning a trivial solution, where Litjens et al. [7] model is
trained to reconstruct a clean input from the corrupted ver-
sion from noise or another corruption which is done by corr-
A Tour of Unsupervised Deep Learning Current Medical Imaging, 2021, Vol. 17, No. 00 3

Fig. (2 A-D). A collage of networks of autoencoders and its variants. (A higher resolution / colour version of this figure is available in the
electronic copy of the article).

Fig. (3). A collage of networks in (A) Convolutional autoencoder, (B) Variational autoencoder, and (C) Contractive autoencoders. (A higher
resolution / colour version of this figure is available in the electronic copy of the article).
4 Current Medical Imaging, 2021, Vol. 17, No. 00
upting the initial input x into by using a stochastic func-
tion . The corrupted input is then mapped to
a hidden representation and recon-
struction z = gθ (y). A schematic representation of DAE is
shown in Fig. (2(C)). Parameters θ and θ' are initialized ran-
domly and trained using stochastic gradient descent to mini-
mize average reconstruction error. The denoising autoen-
coders continue minimizing the same reconstruction loss be-
tween clean X and reconstruction from Y. This continues
maximizing a lower bound on the mutual information be-
tween input x and representation y, and the difference is ob-
tained by applying mapping fθ to a corrupted input. Hence,
such learning is better than the identity, and it extracts fea-
tures useful for denoising. Stack denoising autoencoder (S-
DAE) is a deep network utilizing the power of DAE [6, 8]
and RBMs [9, 10].
3.1.3. Sparse Autoencoder
The limitation of autoencoders to have only small num-
bers of hidden units can be overcome by adding a sparse con-
straint, where a large number of hidden units, more than
one, can be introduced. The aim of sparse autoencoder
(SAE) is to make a large number of neurons have a low aver-
age output so that neurons are inactive most of the time.
Sparsest can be achieved by introducing a loss function dur-
ing the training or by simple manual zeroing few strongest
hidden unit activations. A schematic representation of SAE
is shown in Fig. (2(D)).
If the activation function of the hidden neurons is aj, the
average activation function of each hidden neuron j will then
be given by:
(4)
The objective of sparse constraints is to minimize Pj so
that Pj = P, where P is a sparse constraint very close to 0,
such as 0.05. To enforce sparse constraints, a penalty term is
added to the cost function, which penalizes j, de-weighting
significantly from P. The penalty term is the Kullback-Lei-
bler (KL) divergence between Bernoulli random variables,
which can be calculated as [11, 12].
(5)
where N2 is the number of neurons in the hidden layers,
and index j is summing over the hidden units in the network.
(6)
Raza and Singh
The property of the penalty function is that KL (P || j) =
0, if Pj = j, otherwise, it increases gradually as j diverse
for P.
3.1.4. Convolutional Autoencoder
As current research witness, the stacked AE is the most
popular and widely used network model in deep unsuper-
vised architecture [7], which requires layer-wise pre-train-
ing. When layers go deeper during the pre-training process
making fully connected layers, the entire process becomes
very tedious and time-consuming. Li et al. [13] proposed the
first trial to train convolutional directly in an end-to-end
manner without pre-training. Guo et al. [14] suggested that
convolutional autoencoder (CAE) is beneficial to learn fea-
tures for images and preserving the local structure of data
and to avoid distortion of feature space. A typical architec-
ture of CAE is depicted in Fig. (3).
3.1.5. Variational Autoencoder
Another variant of autoencoder, called variational autoen-
coder (VAE), was introduced as a generative model [15]. A
typical architecture of VAE is given in Fig. 3. They utilize
the strategy of deriving a lower bond estimator from the di-
rected graphical models with a continuous distribution of la-
tent variables. The generative parameter θ in the decoder
(generative model) assists the learning process of the varia-
tional parameter, ϕ as an encoder in the variational approxi-
mation model. VAEs apply the variation approach to latent
representation, learning as additional loss component train-
ing estimators, known as Stochastic Gradient Variational
Bayes (SGVB) and Autoencoding Variational Bayes
(AEVB) [15]. It optimizes the parameter ϕ and θ for the
probabilistic encoder qØ (Z, X), which is an approximation
to the generative model pθ (Z, X) where z is the latent vari-
able and x is a continuous or discrete variable. It aims to
maximize the probability of each x in the training data set un-
der the entire generative process. However, an alternative
configuration of generative latent variable modeling rises to
give deep generative models (DGMs) instead of the existing
assumption of symmetric Gaussian posterior [16].
Recently, Ilse and collaborators [114] extended the VAE
framework, named domain invariant variational autoencoder
(DIVA), to tackle the problem of domain generalization,
such as learning representation from a set of the previously
unseen domain. Considering a perfectly disentangled latent
space, it is hypothesized that there exists a latent subspace
that is invariant to change, called domain invariant, d. The
DIVA considers three independent latent subspaces such as
domain (zd), class (zy), and residual variations (zx). As zd and
zy are marginally independent, the model would learn repre-
sentations zy, which are invariant with respect to the domain
d. Finally, these three latent variants are used by a single de-
coder pθ (x|zd, zx, zy) to reconstruct x. A detailed discussion
on DIVA can be found in [114].
A Tour of Unsupervised Deep Learning Current Medical Imaging, 2021, Vol. 17, No. 00 5

3.1.6. Contractive Autoencoder

The contractive autoencoder was presented by Rifai
(2011) [17] as a novel approach for training deterministic au-
toencoder. It has an additional explicit regularizer in the ob-
jective function that enables the model to learn a function
with the slightest variations in the input values. This additio-
nal regularizer corresponds to the squared Frobenius norm
of the Jacobian matrix of given activation concerning the in-
put. The general architecture of the contractive autoencoder
is given in Fig. (3). The contractive autoencoder is obtained
with the regularization term in the following equation yield
the final objective function.
(7)
The difference between contractive AE and DAE is stat-
ed in earlier work [6]. The latter explicitly encourages ro-
bustness of representation, whereas the former stresses the
robustness of reconstruction. This property makes sense of
contractive AE as a better choice than DAEs to learn useful
feature extraction. Table 1 presents a summary of autoen-
coders and their variants, and Table 2 presents the applica-
tions of autoencoders and their variants for medical image
analysis.

Table 1. Summary of autoencoders and their variants.

Types Descriptions References

Autoencoder One of the simplest forms that aim to learn a representation (encoding) for a set of data [18, 19]
Stacking autoencoder An autoencoder having multiple layers where the outputs of each layer are given as inputs of the successive layer [20]
Sparse autoencoder Encourages hidden units to be zero or near to zero [21]
Denoising autoencoder Capable of predicting true inputs from noisy data [22, 23]
Convolutional autoencoder Learns features, preserves the local structure of data, and avoids distortion of feature space [14]
A generative model utilizing the strategy of deriving a lower bond estimator from directed graphical models with a
Variational autoencoder [15]
continuous distribution of latent variables
Contractive autoencoder Forces the encoder to take small derivatives [17]

Table 2. Applications of autoencoders and their variants for medical image analysis.

Method Task Image Type Remarks References

SAE AD/MCI classification MRI SAE accompanied by supervised fine-tuning [24]
Extraction of latent features on a huge set of features obtained from MRI
SAE AD/MCI/HC classification MRI and PET [25]
and PET images using SAE
SAE AD/MCI/HC classification MRI SAE used to pre-train 3D CNN [29]
SAE MCI/HC classification fMRI SAE used for feature extraction, HMM as a generative model on top [26]
SAE Hippocampus segmentation MRI SAE used for representation learning and measurement of target/atlas patch [30]
SAE used to learn appearance features to steer the shape model for segmen-
SAE Visual pathway segmentation MRI [31]
tation
Uses an ensemble of denoising SAE (pre-trained with RBMs). Denoising
SAE Denoising DCE-MRI MRI [32]
contrast-enhanced MRI sequences using expert DNNs
SAE was used to detect coronavirus disease 2019 (COVID-19) infections
SAE COVID-19 diagnosis CT [121]
from the CT image
SSAE Nucleus detection Digital pathology image Detection of nuclei on breast cancer digital histopathological images [33]
SAE is applied to classify tissues and their subsequent histogram
SAE Stain normalization Digital pathology image [34]
Matching
Unsupervised CNN with SAE to learn features from unlabeled data for
SAE Density classification Mammography [28]
breast texture and density classification
Learn to extract features from multi-parametric MRI data, subsequently cre-
SAE Lesion classification MRI [27]
ates a hierarchical classification to detect prostate cancer
Detection of Heart, kidney, and SAE used for acquisition of Spatio-temporal features on 2D along with time
SAE MRI [5]
liver location DCE-MRI
SAE Cell segmentation Digital pathology image Learning spatial relationships [35]
Segmentation of right ventricle
SAE MRI SAE applied to obtain initial right ventricle segmentation [36]
in cardiac MRI
The SDAE trained with data and their structured labels for cell segmenta-
SDAE Cell segmentation Digital pathology image [37]
tion
A post-processing method that is based on DAE to improve the anatomical
Post-DAE Image segmentation X-ray, MRI [112]
plausibility of image segmentation algorithms
6 Current Medical Imaging, 2021, Vol. 17, No. 00
Method Task Image Type
SSAE AD MRI
SSAE Brain tumor detection MRI
SDAE Breast lesion Ultrasound and CT
SDAE for an unsupervised early prediction of a patient’s future clinical
SDAE Patient clinical events Patient clinical history
SDAE - CT/MRI
DCAE Modeling task fMRI tfMRI
DCAE Tuberculosis severity scoring CT
CAE AD/MCI/HC classification fMRI
Sparse CAE to detect and encode nuclei and feature extraction from tissue
CAE Nucleus detection Digital pathology image
CAE was used to extract features from unlabeled data for a CNN-based 3D
CAE 3D liver segmentation CT
[Abbreviations: CAE: Convolutional autoencoder; DCAE: Deep convolutional autoencoder; SAE: Sparse autoencoder; SSAE: Stacked sparse autoencoder; SDAE: Stack denoising
autoencoder; DAE: Deep autoencoders; H&E: hematoxylin and eosin staining; AD: Alzheimer’s disease; MCI: Mild cognitive impairment; fMRI: Functional magnetic resonance
imaging; sMRI: Structural magnetic resonance imaging; rs-fMRI: Resting-state fMRI; EEG: electroencephalography; DBN: Deep belief network; RBM: Restricted Boltzmann
machine].
SAE were used by Suk and Shen [24] and Suk et al. [25]
to extract low-level, latent features from multimodal imag-
ing datasets of MCI, AD, and HC (healthy control subjects).
MRI, FDG-PET, and biological marker data from CSF were
pre-trained on SAE followed by fusion of extracted latent
features with multi-kernel SVM to yield the diagnosis. Su-
pervised fine-tuning of the whole network using the pre--
trained parameters at the starting point has led to the consid-
erable accuracy of the output. Another variant of SAE, name-
ly Deep Autoencoders (DAE), when used along with the
Hidden Markov Model (HMM), renders the diagnosis of
MCI based on rs-fMRI data as input [26]. Apart from its use
in classification, prediction, and diagnosis of various brain
disorders, several studies use SAE for lesion classification in
prostate cancer, texture, and density of breast cancer, detec-
tion of visceral organ’s location from DCE MRI images, and
so on [5, 27, 28].
Besides the autoencoders and their several variants men-
tioned in Table 1, there are many other variants of autoen-
coders, although not prominently visible. However, they are
recently proposed and their applications in various aspects
of medical image analysis are shown. For instance, Super-
vised Switching Autoencoders (SSAs) proposed for
Alzheimer’s disease classification from the single-slice im-
age and disease regional analysis [116], Spatiotemporal At-
tention Autoencoder (STAAE) for attention deficit hyperac-
tivity disorder (ADHD) classification [117], Disentangled
Autoencoder for cross-stain feature extraction [118], 3D resi-
dual autoencoder (3D ResAE) to model deep representations
of fMRI [119], and Dense Residual Convolutional Auto En-
coder (DRCAE) for retinal blood vessels segmentation
[120].
3.2. Restricted Boltzmann Machines
Restricted Boltzmann Machines (RBMs) are a variant of
Markov Random Field (MRF), constituting single layer undi-
rected graphical model with an input layer or visible layer x
= (x1, x2...... xN) and a hidden layer h = {h1, h2, ….HM}. The
connection between nodes/units is bidirectional, so each giv-
Raza and Singh
Remarks References
SSAE for early detection of Alzheimer’s disease from brain MRI [38]
A two-layer SSAE for the detection of brain tumors from MRI slices [115]
Stacked Denoising AE for Diagnosis of breast nodules and lesions [39]
[40]
events and disease
Multi-modal SDAE is used to pre-train the DNN [41]
Deep Convolutional AE to model tfMRI [42]
DCAE is used for descriptors extraction from a 3D CT image [113]
CAE used to pre-train 3D CNN [43]
[44]
section images
[111]
image segmentation
en input vector x can take the latent feature representation h
and vice-versa. These are generative models that learn a
probability distribution over the given input space and gener-
ates a new data point [45]. An illustration of a typical RBM
is shown in Fig. (4(A)). In fact, RBMs are restricted ver-
sions of Boltzmann machines where neurons must form an
arrangement of bipartite graphs. Due to this restriction, pairs
of nodes belonging to each of the visible and hidden nodes
have a symmetric connection between them, and nodes
within a group have no internal connections. This restriction
makes RBM a more efficient training algorithm than the gen-
eral case of the Boltzmann machine. Hinton et al. (2010)
[45] proposed a practical guide to train RBMs.
RBMs have been utilized in various aspects of medical
image analysis as classification, segmentation, or detection
of objects from radio images or pathological data. Yoo et al.
(2014) [46] proposed a method for segmentation of multiple
sclerosis (MS) lesion in multi-channel 3D MRI images.
Huang et al. 2016 [47] applied the RBM for Blind Source
Separation (BSS) for fMRI data instead of Independent Com-
ponent Analysis (ICA) to detect latent sources that differenti-
ate internal or functional interaction in the special brain re-
gion. Cai et al. (2016) [48] forwarded a novel architecture
called Transformed Deep Convolutional Network (TDCN),
a multi-output Convolutional Restricted Boltzmann Machine
(CRBM) implemented to fuse the learning features from dif-
ferent modalities, e.g., CT and MRI in an unsupervised man-
ner which improved the recognition of vertebra pattern of
different species. Jaumard-Hakoun et al. (2016) [49] trained
a deep autoencoder in two phases. In the first phase, Transla-
tional RBM (tRBM) learned the relationship between inputs
(US and contour) data and, in the second phase, it recon-
structed the contour of the tongue motion during a speech
from US image. However, RBMs and their variant were
widely applied for classification problems. Cao et al. (2015)
[50] solved the classification of imbalanced data in breast
mammogram based oversampling and semi-supervised learn-
ing model for false-positive reduction. Zhang et al. (2016)
[51] achieved a classification AUC of 93.40% to differenti-
ate between benign and malignant breast tumors. Van
A Tour of Unsupervised Deep Learning Current Medical Imaging, 2021, Vol. 17, No. 00 7

Fig. (4). Various unsupervised learning models (A) Restricted Boltzman Machine, (B) Deep Belief Network, (C) Deep Boltzman Machine,
(D) Generative Adversarial Network. (A higher resolution / colour version of this figure is available in the electronic copy of the article).

Table 3. Applications of RBM for medical image analysis.

Method Task Image Type Remarks References

Uses multi-channel 3D MR images of multiple sclerosis (MS) lesion for MS seg-
RBM Multiple sclerosis lesions 3DMRI [46]
mentation
RBM based method for oversampling and semi-supervised learning to solve the
RBM Mass detection in breast cancer Mammography [50]
classification of imbalanced data with a few labeled samples
RBM is used for both internal and functional interaction-induced latent source de-
RBM fMRI blind source separation fMRI [47]
tection
RBM Vertebrae localization CT, MRI RBMs to locate the exact position of the vertebrae [48]
Shear wave elastography for classification of benign and malignant mammary
RBM Benign/Malignant classification Ultrasound [51]
gland tumors using RBM
Analysis of tongue motion during the speech, using autoencoders in combination
tRBM Tongue contour extraction Ultrasound [49]
with RBM
Lung tissue classification and airway Discriminative and generative learning by CRBM to develop filters for training as
CRBM CT [52]
detection well as classification
Achieves average recognition accuracy for ventricular and supraventricular ec-
RBM Cardiac arrhythmia classification ECG topic beats (93.63% and 95.57%, respectively) for Cardiac arrhythmia classifica- [53]
tion
RBM Brain lesion segmentation MRI RBM is used for feature learning and a Random Forest as a classifier [54]
A Restricted Boltzmann machine with backpropagation has been used for histo-
RBM Breast-image classification MRI [55]
pathological breast-image classification
Four-layer RBM is trained with the frequency signal of EEG used in brain-com-
RBM Motor imagery classification EEG [84]
puter interface research
[tRBM: Translational RBM; CRBM: Convolutional RBM].
8 Current Medical Imaging, 2021, Vol. 17, No. 00 Raza and Singh

Tulder and de Bruijne (2016) [52] proposed the Convolution-
al RBM, which performed good in both training data and
lung texture classification. Mathews et al. (2018) [53] de-
monstrated single-lead ECG signals classification for detec-
tion of ventricular and supraventricular heartbeats. Pereira et
al. (2017) [54] highlighted a combined RBM-Random forest
classifier which jointly correlated between features imaging
data for brain tumor segmentation and penumbra estimation.
A brief account for the application of RBMs in medical im-
age analysis is shown in Table 3.
3.3. Deep Belief Networks
Deep Belief Networks (DBN) is a kind of deep neural
network proposed by Bengio (2009) [56]. It is a greedy lay-
er-wise unsupervised learning algorithm with several layers
of hidden variables [8]. Layer-wise unsupervised training
helps the optimization and provides weight initialization for
better generalization. In fact, DBN is a hybrid single proba-
bilistic generative model, like a typical RBM. To construct a
deep architecture like SAEs where AEs layers are replaced
by RBMs, DBN has one lowest visible layer V, representing
the state of input data vector and a series of hidden layers h1,
h2, h3, . . . hL. The following function in DBN represents the
joint distribution of visible unit V, hidden layers hl (l = 1,
2…. L):
(8)
When multiple RBMs are stacked hierarchically, an undi-
rected generative model is formed by the top two layers, and
a directed generative model is formed by lower layers. (Fig.
4(B)) illustrates the structure of DBN. Hinton et al. [57] for-
warded an algorithm “wake-sleep” for unsupervised neural
networks. Further, Hinton et al. (2006a) [10] applied a fast
learning algorithm based on layer-wise training procedure,
where lower layers learn low-level features and subsequent-
ly higher layers learn high-level features.
DBN has received a lot of attention in the field of com-
puter vision for success in object recognition. Still, it was
too expensive, like training 3D images due to large training
parameters, so the limitation could not be overcome. Recent-
ly a new variant of DBN was introduced, namely, convolu-
tional DBN (ConvDBNs) [58]. Other work [59] [60] demon-
strated a DBN architecture that learns low-dimensional mani-
fold of brain 3D-MRI that detects different variations in mor-
phological changes and correlates with Alzheimer’s disease
parameters and Lesion Distribution in Multiple Sclerosis.
Plis et al. (2014) [61] and Pinaya et al. (2016) [62] evaluat-
ed deep learning methods to characterize brain networks in
neurocognitive disorders like Huntington’s disease and vari-
ous brain region in Schizophrenia using DBN and super-
vised fine-tuning whereas Ortiz et al. (2016) [63] classified
Mild Cognitive Impairment (MCI) and AD for simpler diag-
nosis and treatment based on Automated Anatomical Label-
ing (AAL) which provides grey matter (GM) image from
each brain region. Carneiro et al. (2012) [64], Carneiro and
Nascimento (2013) [65] built rigid and non-rigid derivatives

Table 4. Applications of DBNs for medical image analysis.

Method Task Image Type Remarks References

Manifold Learning for AD/HC
DBN MRI DBNs with convolutional RBMs for manifold learning [59]
classification
DBM along with convolutional RBM layers to efficiently train DBMs to
DBN, Convolutio-
multiple sclerosis MRI detect morphological changes in the brain in normal as well as disease [60]
nal RBM
conditions
Huntington’s disease and Schi- Evaluation of DBN to estimate brain networks in neurocognitive disor-
DBN MRI [61]
zophrenia ders like Huntington’s disease and Schizophrenia
A group of voting schemes clubbed using an SVM to better classify AD
DBN AD/MCI/HC classification MRI [63]
and MCI from the brain’s 3D gray matter images
A DBN-assisted system, exploiting non-rigid registration, landmarks,
DBN Left ventricle segmentation Ultrasound [64, 65]
and patches to maneuver multi-atlas segmentation
Characterizing differences in morphology of various brain regions in
DBN Schizophrenia/NH classification MRI [62]
schizophrenia using DBN and supervised fine-tuning
Training DBN to extract features from prostate ultrasonography images
DBN Lesion classification Ultrasound [67]
to classify benign and malignant lesions
The combination of DBN and level set method to yield automated seg-
DBN Left ventricle segmentation MRI [66]
mentation of the left ventricle from cardiac cine MRI
Autism spectrum disorders classi- Classifies Autism spectrum disorders (ASDs) in children using rs-fMRI
DBN rs-fMRI, sMRI [68]
fication and sMRI data based on Random Neural Network clustering
Multi-modal re- Risk factor analysis and osteoporosis prediction based on the heteroge-
DBN Bone disease prediction [85]
cords neous electronic health records (EHRs)
Discrete Wavelet Transform (DWT) is combined with DBN to detect
DWT-DBN Glioblastoma tumor detection MRI [122]
Glioblastoma tumor in MRI
A Tour of Unsupervised Deep Learning
which are based on training classifier to automatic segmenta-
tion of the left ventricle of the heart from ultrasound se-
quence data at the same time. Ngo et al. (2017) [66] intro-
duced a combination of DBN and level set model to yield au-
tomated segmentation of the left ventricle from cardiac cine
MRI. Although DBN also trains for the classification of be-
nign and malignant lesions from prostate ultrasonography,
images from other work [67] [68] classify Autism spectrum
disorders (ASDs) in children using rs-fMRI and sMRI data
based on Random Neural Network clustering. Li et al.
(2014) [85] developed a risk analysis framework for osteo-
porosis prediction based on heterogeneous electronic health
records (EHRs). To improve the classification accuracy, dis-
crete wavelet transform (DWT) was combined with a DBN
and applied to detect Glioblastoma tumors from MRI [122].
Summary of DBN application for medical image analysis
presented in Table 4.
3.4. Deep Boltzmann Machine
Deep Boltzmann Machine (DBM) is a robust deep learn-
ing model proposed by Salakhutdinov et al. (2009) [69] and
Salakhutdinov et al. (2012) [70]. They stacked multiple
RBMs in a hierarchal manner to handle ambiguous input ro-
bustly. (Fig. 4(C)) represents the architecture of DBM as a
composite model of RBMs, which clearly shows how DBM
differs from DBN. Unlike DBN, DBM forms an undirected
generative model combining information from both lower
and upper layers, which improves the representation power
of DBMs. Training of layer-wise greedy algorithm for DBM
[71, 72] is calculated by modifying the procedure of DBN.
Salakhutdinov et al. (2015) [71] and Dinggang et al.
(2017) [73] presented a three-layer DBM where the DBM
learns the parameters θ = {w1, w2} the values of the neighbor
layer(s) and probabilities of visible and hidden units are com-
puted using a logistic sigmoidal function. The derivative of
log-likelihood of the observation (V) with respect to the mod-
el parameter (θ) is computed as,
(9)
where Edata[.] denotes data-dependent obtained from visi-
ble units and Emodel[.] denotes data-independence obtained
from the model.
In the case of preceding deep networks (e.g., RBM or
DBN), deep Boltzmann machine has not gained much atten-
Table 5. Applications of DBMs for medical image analysis.
Method Task Image Type
Deep Boltzmann Machine-Driven level set for heart segmentation during radia-
DBM Heart motion tracking MRI
DBMs on multimodal images from MRI and PET scans for disease classifica-
DBM/RBM AD/MCI/HC classification MRI/PET
DBM Medical image retrieval Multi digital image
Binary classification of cancers region using three-layer with backpropagation
DBM Cancers region classification Hyperspectral image
Current Medical Imaging, 2021, Vol. 17, No. 00 9
tion from the research communities in the area of medical
image analysis. Still, with the power of well-built modeling
abilities of DBM, even with a small number of training data,
DBM might get its due place in the future. Suk et al. (2014)
[74] proposed a high-level latent and combined feature repre-
sentation and fusion of multimodal images from MRI and
PET using DBM, which compared the binary classification
of Alzheimer's disease (AD) and have Mild Cognitive Im-
pairment (MCI) disease. Cao et al. (2014) [75] developed a
deep DBM for next-generation medical image revival sys-
tem instead of content-based image retrieval (CBIR), which
learned joint density from the multimodal learning model to
get relative missing modalities. Recently, Wu et al. (2018)
[76] developed a frame-by-frame heart segmentation by em-
ploying a three-layered DBM using cine MRI images to find
local and global features of heart contours. Summary of the
DBM applications for medical image analysis is shown in
Table 5.
3.5. Generative Adversarial Network (GAN)
Generative Adversarial Network (GAN) [77] is one of
the recent promising techniques for building flexible deep
generative unsupervised architecture. Goodfellow et al.
(2014) [77] proposed two models generative model G and
Discriminative model D, where G capture data distribution
(pg) over real data t, and D estimates the probability of a
sample coming from training data (m) not from G. Dur-
ing iterations, the backpropagation generator and the discrim-
inator compete with each other. In the training procedure,
the probability of D is maximized. This framework func-
tions like a mini-max two-player game. The value function
V (G, D) establishes following two-player mini-max game is
given by:
(10)
where D(t) represents the probability of t from data m
and pdata is the distribution of real-world data. This model
seems to be stable and improved as pg = pdata. A typical archi-
tecture of GAN is depicted in Fig. (4(D)). In fact, these two
adversaries, Generator and Discriminator, continuously bat-
tle during the processing of training. GAN has been applied
to generate samples of photorealistic images to visualize
new designs.
Remarks References
[76]
tion therapy of cancer patients on cine MRI images
[74]
tion
DBM based multi-model learning to learn joint density model [75]
[83]
architecture
10 Current Medical Imaging, 2021, Vol. 17, No. 00 Raza and Singh

The recent applications of GAN produce promising re-
sults in many diverse fields of medical image analysis. Hu et
al. (2017) [78] employed conditional generative adversarial
networks (cGAN) to simulate ultrasound images at given 3D
spatial locations, while Bi et al. (2017) [79] designed Multi-
-channel GAN (M-GAN) to synthesize PET data which im-
prove the AUC of PET-based CAD system. Bi et al. (2018)
[80] used an improved GAN, which uses dual-path adver-
sarial learning for Fully Convolutional Network (FCN)
based image segmentation of the region of interest (ROI)
without any medical image specification. Iqbal and Ali
(2018) [81] and Canas et al. (2018) [82] proposed MI-GAN,
which generates precise segmented images for the applica-
tion of supervised learning of retinal images, retaining patho-
logical quality. Wang et al. (2018) [83-87] proposed 3D-con-
ditional GAN synthesizing high-quality brain region PET im-
age from low-dose PET image for treatment of MCI disease.
In several applications of GAN and its variants, medical im-
age generation or augmentation of various modalities such
as CT, MRI, X-ray, and PET was carried out to maximize
the size of the training dataset with realistic, high-quality im-
ages and preserving all deciding features in synthetic images
which largely impact on the performance of segmenting
ROI or analysis [88-99]. Some of the further applications of
cGAN include brain tumor segmentation from MRI [100],
tissue segmentation from Ultrasound [101], retinal image
synthesis [98], and effectively generate high-quality and real-
istic coronavirus diseases 2019 (COVID-19) CT images
[130]. Zhang et al. (2020) [127] proposed a task-driven
GAN (TD-GAN) in order to achieve both synthesis and pars-
ing simultaneously for unseen real X-ray images without
any annotations requirement from the X-ray image domain.
The TD-GAN model provides a promising average dice of
86%, which achieves a similar level of accuracy as super-
vised training. Few other applications of GAN include a re-
current GAN (RNN-GAN) to mitigate data imbalance
problems in medical image semantic segmentation [128], a
GAN with dual discrimination (DD-GAN) to improve the
recognition in skin lesion segmentation [129], and so on.
There are several recent reviews on GAN and its
variants in medical image analysis, including medical image
generation [102], mammogram analysis [123], and medical
imaging and analysis [124-126]. A summary of the GAN ap-
plications for medical image analysis is presented in Table
6.

Table 6. Applications of GAN for medical image analysis.

Method Task Image Type Remarks References

MI-GAN generates precise segmented images for the application of super-
GAN Synthesis of retinal images Retinal [81]
vised learning of retinal images
GAN used to produce photorealistic images which retain pathological
GAN Chest X-ray X-ray [82]
quality
Dual GAN- Segmentation of regions of in- Improve GAN using dual-path adversarial learning for Fully Convolution-
- [80]
FCN terest (ROIs) al Network-based image segmentation
Treatment of lymphomas and
GAN PET Multi-channel GANs are used to synthesize PET data [79]
lung cancer
Simulation of B-mode ultra- Conditional GANs are used to simulate ultrasound images at given 3D spa-
GAN Ultrasound [78]
sound images tial locations
Least-squares GAN used to model the low dimension image to high-quali-
GAN Reconstruction MRI [86]
ty images
Mild cognitive impairment 3D-conditional GAN for generating high-quality PET image from the low
GAN PET [87]
treatment dose PET image
GAN based denoising for improving the quality of the reconstructed
GAN Reconstruction CT [88]
image
Multiphase coronary angio- Cycle-consistent GAN based denoising of low-dose CT image to equiva-
GAN CT [89]
graphy lent routine CT images
Deep convolutional GAN network used to generate a synthetic image of le-
DCGAN Liver lesion classification CT [90]
sion region which improve the classification performance
Utilize unsupervised learning to generate realistic fake lung nodules which
DCGAN Lung cancer diagnosis CT [91]
easily discriminate between benign and malignant lung nodules
Semi-supervised learning for segmenting labeled and unlabeled 3D multi-
GAN Brain MRI segmentation MRI [92]
modal image
Chest pathology classifica-
GAN X-ray GAN framework used for augmentation of an artificial X-ray image [93]
tion
GAN based semi-supervised learning methods used for classification of
GAN Chest X-ray classification X-ray [94]
cardiac abnormalities in chest X-ray
Images (ISIC 2017 Laplacian GAN (LAP-GAN) framework utilize to generate high-resolu-
LAP-GAN Skin lesion synthesis [95]
dataset) tion realistic skin lesion image
An unsupervised learning framework allows blood vessel segmentation in
GAN Retinal vessel segmentation Retinal [96]
retinal fundus images under extremely low annotation
A Tour of Unsupervised Deep Learning Current Medical Imaging, 2021, Vol. 17, No. 00 11

Method Task Image Type Remarks References

GAN and autoencoder based framework for generating synthetic retinal
GAN + VAE Retinal image synthesis Retinal [97]
image and perform precise retinal vessel segmentation
Synthesizing realistic retinal image using conditional GAN (cGAN) and
cGAN Retinal image synthesis Retinal [98]
preserving the same structured annotation
Generating synthetic brain tumors images to achieve precise segmentation
GAN Brain tumor segmentation MRI [99]
of ROI
GAN framework used for MRI brain tumor image augmentation to in-
cGAN Brain tumor segmentation MRI [100]
crease the size of training data
GAN approach for synthetic ultrasound image generation and simulation
cGAN Tissue segmentation Ultrasound of pathological ultrasound images to preserve the intensities regarding spe- [101]
cific tissue
cGAN applied to effectively generate high-quality and realistic
cGAN Chest CT synthesis CT [130]
COVID-19 CT images
A task-driven GAN (TD-GAN) was built to achieve both synthesis and
TD-GAN Image segmentation X-ray parsing simultaneously for unseen real X-ray images without any annota- [127]
tions by the X-ray radiologists
Images (ACD-
C-2017, HVSM- A recurrent GAN (RNN-GAN) was proposed to mitigate the data imbal-
RNN-GAN Image segmentation [128]
R-2016, LiTS-2017 ance problems in medical image semantic segmentation
datasets)
Images (ISIC Skin Le-
A dual discrimination GAN (DD-GAN) was proposed to improve the
DD-GAN Skin lesion segmentation sion Challenge Da- [129]
recognition in skin lesion segmentation
tasets 2017 and 2018)

Table 7. Strengths and weaknesses of unsupervised deep learning models.

Unsupervised Models Strength Weakness

AEs minimize the reconstruction errors between input and output values Presence of a few hidden layer network force it to
Autoencoders (AEs) and
for both linear and non-linear transformation, whereas Stacked AEs ex- learn compressed representation. A simple AE is very
Stacked autoencoder
tend the benefits of a deep network prone to identity learning
DAEs are capable of reconstructing a clean input from a noisy or cor-
Denoising autoencoder The problem of over-generalization or fooling during
rupted version of it and address the risk of learning from identity func-
(DAE) training
tion
SAEs use a simple regularization method where the introduced weight
Sparse autoencoder (SAE) of autoencoder is penalized in the cost function and maintains the com- -
plexity of the model by preventing over-fitting
CAEs are highly scalable, which trains the model directly end-to-end
Convolutional autoencoder
manner without pre-training, which reduces the time required by layer- Training is very complex and difficult
(CAEs)
wise pre-training in Stacked AEs
Due to injected noise, imperfect reconstruction, and the
Variational autoencoder VAE inherits the architecture of autoencoder with the extension of la-
use of direct mean squared error, the generated images
(VAE) tent representation learning
are likely to be more blurry
Contractive autoencoder Contractive AE explicitly encourages robustness of representation by ad- The trade-off between reconstruction error and a regu-
(CAE) ding an explicit regularizer in the objective function larization term
Restricted Boltzmann RBMs are more efficient than a simple Boltzmann machine and learn a Only one layer of the hidden unit and no connection be-
Machines (RBMs) probability distribution over given input tween hidden units
DBN consists of the stacking of multiple RBMs hierarchically and
Deep Belief Networks During the learning network more vulnerable to gener-
trained in a greedy manner so, it extends the strength of RBM for better
(DBN) ate the observed data
generalization
Unlike DBNs, DBMs symmetrically connected binary units form an
Deep Boltzmann machine Only approximate maximum likelihood learning, limit
undirected generative model can learn complex input task, which im-
(DBM) the speed of performance and functionality
proves the representation power of DBM

The biggest problem for GAN to learn implicit struc-

GAN is a flexible deep generative unsupervised architecture; produces a ture in object class and data structure like 2D or 3D im-
Generative Adversarial Net-
deeper understanding of data by generating new fake data and learning age data has complex variation. Also, the difficulty of
work (GAN)
through it training, evaluation of generated samples, and theoreti-
cal limitations (Suganuma et al., 2018)
12 Current Medical Imaging, 2021, Vol. 17, No. 00 Raza and Singh

Table 8. List of software tools/packages for unsupervised learning models.

S. Tools/ Packages
Models/Methods Description Language/Technology URL
No. Name
Deep learning APIs for Java hav-
1 deeplearning4j Autoencoders ing an implementation of several Java https://deeplearning4j.org/
deep learning techniques
A scientific computing framework
with good support for machine
learning algorithms that puts
2 unsupundertorch7 Autoencoder, etc. GPUs first. Unsup package pro- Lua https://github.com/torch/torch7
vides few unsupervised learning al-
gorithms such as autoencoders,
clustering, etc.
MIT licensed deep learning frame-
work that runs on CPU or GPUs
https://github.com/andersbll/deeppy
3 DeepPy Autoencoders and implements autoencoders, in Python
http://andersbll.github.io/deeppy-website/
addition to other supervised learn-
ing algorithms
Build a stacked autoencoder in the
R environment for pre-training of
https://rdrr.io/cran/SAENET/man/SAENET.-
4 SAENET.train Stacked autoencoder feed-forward NN and dimension R package
train.html
reduction of features

Gaussian Mixture Convolutional

Convolutional au- Autoencoder (GMCAE) used for Python, Keras, Ten-
5 kdsb17 https://github.com/alegonz/kdsb17
toencoder CT lung scan using Keras/TensorF- sor-flow-gpu
low
Training a deep autoencoder for http://www.cs.toronto.edu/~hinton/code/Autoe
6 autoencoder Deep autoencoder Matlab
MNIST digits datasets ncoder_Code.tar
Parallelized implementations of
many supervised and unsupervised
machine learning algorithms, in-
7 H2O Deep autoencoder R package https://cran.r-project.org/web/packages/h2o/
cluding GLM, GBM, RF, DNN,
K-Means, PCA, Deep AE, etc.

Domain invariant
Implementation of DIVA, an ex- Python, Scikit-image, https://github.com/AMLab-Amsterdam/DIVA
8 DIVA variational autoen-
tension of variational autoencoder Scikit-learn s
coder (DIVA)
Deep belief network pre-train in
an unsupervised manner with https://rdrr.io/github/TimoMatzen/RBM/sr-
9 Dbn DBN R package
stacks of RBM, which in return c/R/DBN.R
fine-tuned DBN
Restricted Boltzmann machine,
10 darch DBN, RBM deep belief network implementa- R package https://github.com/maddin79/darch
tion
DBN, RBM, deep Implementation of RBM, DBN, https://cran.r-project.org/web/packages/deep-
11 deepnet R package
autoencoders deep-stacked autoencoders net/
DBN and other deep learning im-
12 Vulpes DBN Visual Studio https://github.com/fsprojects/Vulpes
plementation in F#
RBM and DBM are implemented
13 pydbm DBM/ RBM in Python for pre-learning or di- Python https://pypi.org/project/pydbm/
mension reduction
Simple RBM implementation in https://github.com/echen/restricted-boltzman-
14 RBM RBM Python
Python n-machines
RBM and its Implementation of RBM and its
15 xRBM Python https://github.com/omimo/xRBM
variants variants in Tensorflow
Unsupervised representation learn-
16 DCGAN.torch GAN ing using Deep Convolutional Lua https://github.com/soumith/dcgan.torch
GAN
A Tour of Unsupervised Deep Learning Current Medical Imaging, 2021, Vol. 17, No. 00 13

S. Tools/ Packages
Models/Methods Description Language/Technology URL
No. Name
Conditional Adversarial Networks
17 pix2pix GAN for Image-to-image translation syn- Linux Shell Script https://github.com/phillipi/pix2pix
thesizing from the image
Energy-based GAN equivalent to
probabilistic GANs produces high- https://github.com/eriklindernoren/PyTorch-
18 ebgan GAN Python
-resolution images GAN/tree/master/implementations/ebgan

[Abbreviations: ADNI: Alzheimer’s Disease Neuroimaging Initiative; ABIDE: Autism Brain Imaging Data Exchange; DICOM: Digital Imaging and Communications in Medicine;
BCDR: Breast Cancer Digital Repository; CIVM: Center for in vivo Microscopy; DDSM: Digital Database for Screening Mammography; DRIVE: Digital Retinal Images for Vessel
Extraction; IDA: Image & Data Archive; ISDIS: International Society for Digital Imaging of the Skin; NBIA: National Biomedical Imaging Archive; OASIS: Open Access Series of
Imaging Studies; TCGA: The Cancer Genome Atlas; TCIA: The Cancer ImagingArchive].

Table 9. List of benchmark medical image datasets.

S.
Data Set Modalities Medical Condition Accessibility URL
No.
1 ABIDE MRI Autism spectrum disorder Open access http://fcon_1000.projects.nitrc.org/indi/abide/
2 ADNI MRI Alzheimer’s disease Paid http://adni.loni.usc.edu/data-samples/access-data/
3 BCDR Mammography Breast cancer Open access https://bcdr.eu/
Histology of the Embryonic
4 CIVM 3D-MRM Limited access http://www.civm.duhs.duke.edu/devatlas/
and Neonatal Mouse
5 DDSM Mammography Breast cancer Open access http://marathon.csee.usf.edu/Mammography/Database.html
A huge database of various
6 DermNet Photo dermatology Limited access http://www.dermnet.com/
skin diseases
A variety of medical images,
7 DICOM MRI, CT, etc. Open access https://www.dicomlibrary.com
videos, and signal files
Retinal blood vessel segmenta-
2D color images of
8 DRIVE tion to study diabetic retino- Open access http://www.isi.uu.nl/Research/Databases/DRIVE/download.php
the retina
pathy
An online resource for neuros-
9 IDA Open access https://ida.loni.usc.edu/
cience images
Dermoscopy,
10 ISDIS telemedicine, spec- Skin disease Paid https://isdis.org/
troscopy, etc.
Online database of medical im-
Variety of imaging
11 MedPix ages, teaching cases, and clini- Open access https://medpix.nlm.nih.gov
data
cal topics
A database of the National Can-
cer Institute proving medical Limited/ open ac-
12 NBIA CT, PET, MRI, etc. https://imaging.nci.nih.gov/
images of various conditions cess
and anatomical sites
Normal aging or mild to moder-
13 OASIS MRI and PET Open access http://www.oasis-brains.org/
ate Alzheimer's Disease
Collection of MRI, A multimodal image archive Limited/ open ac-
14 TCIA http://www.cancerimagingarchive.net/
CT, etc. for various types of cancer cess
Histopathology slide images
Histopathology slide
15 TCGA from sample portions of vari- Open https://cancergenome.nih.gov/
images
ous types of cancers
Prostate MR Image Segmenta-
16 PROMISE12 MRI Open https://promise12.grand-challenge.org/
tion

4. COMPARISON AMONG UNSUPERVISED DEEP 5. LIST OF SOFTWARE TOOLS/PACKAGES AND

LEARNING MODELS
So far, we have discussed the working and applications
of various unsupervised deep learning models in the field of
medical image analysis. Each model has its own strengths
and weaknesses, as presented in Table 7.
BENCHMARK DATASETS
A plethora of software tools and packages implementing
unsupervised learning models (as discussed in the paper) has
been developed and made available to the research communi-
ty and data analysts. These software tools and packages are
very handy to be used by the researchers. These are avail-
14 Current Medical Imaging, 2021, Vol. 17, No. 00
able in various programming language environments, includ-
ing R, Python, Lua, Java, Matlab, and Linux shell script. In
the last few years, several image datasets have been made
available to the research communities which are used as the
benchmark datasets. Some of the tools/packages and medi-
cal image benchmark datasets are listed in Table 8 and 9, re-
spectively.
6. DISCUSSION, OPPORTUNITIES, AND CHAL-
LENGES
Medical imaging and diagnostic techniques are the most
widely used approaches for the early detection, diagnosis,
and treatment of complex diseases. After significant ad-
vancements in machine learning and deep learning (both su-
pervised and unsupervised), there is a paradigm shift from
the manual interpretation of medical images by human ex-
perts such as radiologists and physicians to automated analy-
sis and interpretation, called computer-assisted diagnosis
(CAD). As unsupervised learning algorithms can derive in-
sights directly from data, we can use them for data-driven de-
cision making. They are more robust, and hence, can be util-
ized as the holy grail of learning and classification
problems. Furthermore, these models are also utilized for
other important tasks, including compression, dimensionali-
ty reduction, denoising, super-resolution, and some degree
of decision making.
Unsupervised learning and CAD are both in their infan-
cy. Researchers and practitioners have many opportunities
in this area. Some of them are: (i) allows exploratory analy-
sis of data, (ii)allows preprocessing for the supervised algo-
rithm when it is used to generate a new representation of da-
ta which ensures learning accuracy and reduces memory
time overheads, (iii) the recent development of cloud com-
puting, GPU-based computing, parallel computing and its
cheaper cost allows big data processing, image analysis, and
execution of complex deep learning algorithms very easily,
(iv) provides seamless and secured smart health care system
at reasonable bandwidth speed [105-108]. Ghoneim et al.
contributed a system for medical image forgery detection us-
ing support vector machine-based classifier [103].
Some of the challenges and research directions are:
(i) Difficult to evaluate whether the algorithm has
learned anything useful: Due to the lack of labels in unsu-
pervised learning, it is nearly impossible to quantify its accu-
racy. For instance, how can we assess whether the K-means
algorithm found the right clusters? In this direction, there is
a need to develop algorithms that can give an objective per-
formance measure in unsupervised learning.
(ii) Difficult to select the right algorithm and hard-
ware: Selection of the right algorithm for a particular type
of medical image analysis is not a trivial task because the
performances of the algorithm are highly dependent on the
types of data. Similarly, hardware requirement also varies
from problem to problem.
(iii) Will unsupervised learning work for me?: It is a
frequently asked question, but its answer depends on the
problem at hand. In the image segmentation problem, the
Raza and Singh
clustering algorithm will only work if the images fit into nat-
urals groups.
(iv) Not a common choice for medical image analysis:
Unsupervised learning is not a common choice for medical
image analysis. However, from the literature, it is revealed
that these (autoencoders and their variants, DBN, RBM,
etc.) are mostly used to learn the hierarchy level of features
for classification tasks. It is expected that unsupervised learn-
ing will play a pivotal role in solving complex medical imag-
ing problems that are not only scalable to a large amount of
unlabeled data but also suitable for performing unsupervised
and supervised learning tasks simultaneously (Yi et al.,
2018).
(v) Development of patient-specific anatomical and
organ model: Anatomical skeletons play a crucial role in un-
derstanding diseases and pathology. Patient-specific anatomi-
cal model is frequently used for surgery and interventions.
They help to plan procedures, perform measurements for de-
vice surging, and predict the outcome of post-surgery com-
plexities. Hence, the algorithm needs to be developed to con-
struct a patient-specific anatomical and organ model from
medical images.
(vi) Heterogeneous image data: In the last two to three
decades, more emphasis was given to well-defined medical
image analysis applications, where developed algorithms
were validated on well-defined types of images with wel-
l-defined acquisition protocol. The algorithms are required,
which can work on more heterogeneous data.
(vii) Semantic segmentation of images: Semantic seg-
mentation is the task of complete scene understanding, lead-
ing to knowledge inference from imagery. Scene understand-
ing is a core of computer vision problems with several appli-
cations, including human-computer interaction, self-driving
vehicles, virtual reality, and medical image analysis. For ins-
tance, MRI prostate segmentation using deep supervised
CNN and boundary-weighted neural network-based segmen-
tation was carried out earlier [108, 110]. In another study, a
bidirectional convolution targets all slices of 3D volume da-
ta instead of an individual slice [104]. The semantic segmen-
tation of medical images with acceptable accuracy is still
challenging.
(viii) Medical video transmission: Enabling 3D video
in recently adopted telemedicine and U-healthcare applica-
tions result in more natural viewing conditions and better di-
agnosis. Also, remote surgery can be benefited from 3D
video because of the additional dimensions of depth. Howev-
er, it is crucial to transmitting data-hungry 3D medical video
streams in real-time through limited bandwidth channels.
Hence, efficient encoding and decoding techniques for 3D
video data transmission are required.
(ix) Need extensive inter-organizational collabora-
tions: Inter-professional and inter-organizational collabora-
tion is important for better functioning of the healthcare sys-
tem, eliminating some of the pitfalls such as limited re-
sources, lack of expertise, aging populations, and combating
chronic diseases (Karam et al., 2017). Medical image-based
A Tour of Unsupervised Deep Learning
CAD needs extensive inter-organizational collaborations
among doctors, radiologists, medical image analysts, and
computational data analysts.
(x) Need to capitalize on the big medical imaging mar-
ket: According to the IHS Market report (https://technolo-
gy.ihs.com.), the medical imaging market has a total global
revenue of $21.2 billion in 2016, which is forecasted to
touch $24.0 billion by 2020. According to WHO, the global
population will rise from 12% to 22% from 2015 to 2050.
Population aging leads to an increased rate of chronic diseas-
es globally, and hence there is a need to capitalize on a big
medical imaging market worldwide.
(xi) Black-box and its acceptance by health professio-
nals: Machine learning algorithms are boons which solve
the problems earlier thought to be unsolvable. However, it
suffers from being “black-box”, i.e., how output arrives
from the model is very complicated to interpret. Particularly,
deep learning models are almost non-interpretable but are
still used for complex medical image analysis. Hence, its ac-
ceptance by health professionals is still questionable.
(xii) Will technology replace radiologists? For the pro-
cessing of medical images, deep learning algorithms help to
select and extract important features and construct new ones,
leading to a new representation of images not seen before.
For the image interpretation side, deep learning helps to iden-
tify, classify, and quantify disease patterns. It also allows the
measurement of predictive targets, makes predictive models,
and so on. So, will technology “replace radiologists”, or mi-
grate to “virtual radiologist assistant” in the near future?
Hence, the following slogan is quite relevant in this context:
“Embrace it, it will make you stronger; reject it, it may make
you irrelevant”.
In a nutshell, unsupervised learning is an open topic
where researchers can make contributions by developing a
new unsupervised method to train the network (e.g., solve a
puzzle, generate image patterns, image patch comparison,
etc.). Re-thinking of creating a great unsupervised feature
representation (e.g., what is the object and what is the back-
ground?) is nearly analogous to the human visual system.
CONCLUSION
Medical imaging is one of the important techniques for
the early detection, diagnosis, and treatment of complex dis-
eases. Interpretation of medical images is usually performed
by human experts such as radiologists and physicians. After
the success of machine learning techniques, including deep
learning and the availability of cheap computing infrastruc-
ture through cloud computing, there has been a paradigm
shift in the field of computer-assisted diagnosis (CAD).
Both supervised and unsupervised machine learning ap-
proaches are widely applied in medical image analysis, each
of them with their pros and cons. Due to the fact that human
supervisions are not always available or can be inadequate
or biased, therefore, unsupervised learning algorithms, in-
cluding their deep architecture, provide hope for the future
with lots of advantages.
Current Medical Imaging, 2021, Vol. 17, No. 00 15
Unsupervised learning algorithms derive insights direct-
ly from data and use them for data-driven decision making.
Unsupervised models are more robust and they can be util-
ized as the holy grail of learning and classification
problems. These models are also used for other tasks includ-
ing compression, dimensionality reduction, denoising, su-
per-resolution, and some degree of decision making. There-
fore, it is better to construct a model without knowing what
tasks will be at hand and whether or not we would use repre-
sentation (or model). All in all, we can think of unsuper-
vised learning as a preparation (pre-processing) step for su-
pervised learning tasks, where unsupervised learning of rep-
resentation may allow better generalization of a classifier.
CONSENT FOR PUBLICATION
Not applicable.
FUNDING
None.
CONFLICT OF INTEREST
The authors declare no conflicts of interest, financial or
otherwise.
ACKNOWLEDGEMENTS
None.
REFERENCES
[1] Wani N, Raza K. Multiple Kernel-Learning Approach for Medical
Image Analysis. 2018.
http://dx.doi.org/10.1016/B978-0-12-813087-2.00002-6
[2] Jabeen A, Ahmad N, Raza K. Machine Learning-Based State-of-
the-Art Methods for the Classification of RNA-Seq Data. Classifi-
cation in BioApps 2018; 6: 133-72.
http://dx.doi.org/10.1007/978-3-319-65981-7_6
[3] Bourlard H, Kamp Y. Auto-association by multilayer perceptrons
and singular value decomposition. Biol Cybern 1988; 59(4-5):
291-4.
http://dx.doi.org/10.1007/BF00332918 PMID: 3196773
[4] Bengio Y, Courville A, Vincent P. Representation learning: a re-
view and new perspectives. IEEE Trans Pattern Anal Mach Intell
2013; 35(8): 1798-828.
http://dx.doi.org/10.1109/TPAMI.2013.50 PMID: 23787338
[5] Shin HC, Orton MR, Collins DJ, Doran SJ, Leach MO. Stacked
autoencoders for unsupervised feature learning and multiple organ
detection in a pilot study using 4D patient data. IEEE Trans Pat-
tern Anal Mach Intell 2013; 35(8): 1930-43.
http://dx.doi.org/10.1109/TPAMI.2012.277 PMID: 23787345
[6] Vincent P, Larochelle H, Lajoie I. Stacked denoising autoen-
coders: learning useful representations in a deep network with a lo-
cal denoising criterion. J Mach Learn Res 2010; 11: 3371-408.
[7] Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning
in medical image analysis. Med Image Anal 2017; 42: 60-88.
http://dx.doi.org/10.1016/j.media.2017.07.005 PMID: 28778026
[8] Bengio Y, Lamblin P, Popovici D, et al. Greedy layer-wise train-
ing of deep networks. Adv Neural Inf Process Syst 2006; 19:
153-60.
[9] Hinton GE, Salakhutdinov RR. Reducing the dimensionality of da-
ta with neural networks. Science 2006; 313(5786): 504-7.
http://dx.doi.org/10.1126/science.1127647 PMID: 16873662
[10] Hinton GE, Osindero S, Teh YW. A fast learning algorithm for
deep belief nets. Neural Comput 2006; 18(7): 1527-54.
http://dx.doi.org/10.1162/neco.2006.18.7.1527 PMID: 16764513
[11] Ng A. Sparse autoencoder lecture notes 2013.web.stanford.edu/-
16 Current Medical Imaging, 2021, Vol. 17, No. 00
class/cs294a/sparseAutoencoder.pdf
[12] Makhzani A, Frey B. k-Sparse Autoencoders. arxiv: preprint
2013.
[13] Li F, Qiao H, Zhang B. Discriminatively boosted image clustering
with fully convolutional auto-encoders. Pattern Recognit 2018;
83: 161-73.
http://dx.doi.org/10.1016/j.patcog.2018.05.019
[14] Guo X, Liu X, Zhu E, Yin J. Deep clustering with convolutional
autoencoders. International Conference on Neural Information Pro-
cessing. 373-82.
[15] Kingma and Max Welling. Auto-encoding variationalbayes.
CoRRabs 2013.
[16] Partaourides H, Chatzis SP. Asymmetric deep generative models.
Neurocomputing 2017; 241: 90.
http://dx.doi.org/10.1016/j.neucom.2017.02.028
[17] Rifai S, Vincent P, Muller X, et al. Contractive auto-encoders: ex-
plicit invariance during feature extraction. Proceedings of the 28th
International Conference on International Conference on Machine
Learning (ICML 2011). 833-40.
[18] Ballard DH. Modular Learning in Neural Networks.AAAI. 1987;
pp. 279-84.
[19] Pinaya WHL, Sandra V, Rafael G-D, et al. Autoencoders Machine
Learning Academic Press. 2020; pp. 193-208.
[20] Zabalza J, Ren J, Zheng J, et al. Novel segmented stacked autoen-
coder for effective dimensionality reduction and feature extraction
in hyperspectral imaging. Neurocomputing 2016; 185: 1-10.
http://dx.doi.org/10.1016/j.neucom.2015.11.044
[21] Goodfellow I, Lee H, Le Q, et al. Measuring invariances in deep
networks. Adv Neural Inf Process Syst 2009; 22: 646-54.
[22] Gallinari P, LeCun Y, Thiria S, et al. Memoires associative dis-
tributes. Proceedings of COGNITIVA.
[23] Vincent H, Larochelle Y. Extracting and composing robust fea-
tures with denoising autoencoders. In: Cohen WW, McCallum A,
Roweis ST, Eds. Proceedings of the Twenty-fifth International
Conference on Machine Learning (ICML’08). 1096-103.
http://dx.doi.org/10.1145/1390156.1390294
[24] Suk H-I, Shen D. Deep learning-based feature representation for
AD/MCI classification.Proceedings of the Medical Image Comput-
ing and Computer-Assisted Intervention. 8150: 583-90.
[25] Suk H-I, Lee S-W, Shen D. Alzheimer’s Disease Neuroimaging
Initiative. Latent feature representation with stacked auto-encoder
for AD/MCI diagnosis. Brain Struct Funct 2015; 220(2): 841-59.
http://dx.doi.org/10.1007/s00429-013-0687-3 PMID: 24363140
[26] Suk H-I, Wee C-Y, Lee S-W, Shen D. State-space model with
deep learning for functional dynamics estimation in resting-state
fMRI. Neuroimage 2016; 129: 292-307.
http://dx.doi.org/10.1016/j.neuroimage.2016.01.005 PMID:
26774612
[27] Zhu Y, Wang L, Liu M, et al. MRI-based prostate cancer detec-
tion with high-level representation and hierarchical classification.
Med Phys 2017; 44(3): 1028-39.
http://dx.doi.org/10.1002/mp.12116 PMID: 28107548
[28] Kallenberg M, Petersen K, Nielsen M, et al. Unsupervised deep
learning applied to breast density segmentation and mammograph-
ic risk scoring. IEEE Trans Med Imaging 2016; 35(5): 1322-31.
http://dx.doi.org/10.1109/TMI.2016.2532122 PMID: 26915120
[29] Payan A, Montana G. Predicting Alzheimer’s disease: a neuroi-
maging study with 3D convolutional neural networks. arXiv
preprin 2015.
[30] Guo Y, Wu G, Commander L-A, et al. Segmenting hippocampus
from infant brains by sparse patch matching with deep-learned fea-
tures. International Conference on Medical Image Computing and
Computer-Assisted Intervention. 308-15.
http://dx.doi.org/10.1007/978-3-319-10470-6_39
[31] Mansoor A, Cerrolaza JJ, Idrees R, et al. Deep learning guided par-
titioned shape model for anterior visual path- way segmentation.
IEEE Trans Med Imaging 2016; 35(8): 1856-65.
http://dx.doi.org/10.1109/TMI.2016.2535222 PMID: 26930677
[32] Benou A, Veksler R, Friedman A, et al. De-noising of contrast-en-
hanced MRI sequences by an ensemble of expert deep neural net-
works.Deep Learning and Data Labeling for Medical Applica-
tions. Cham: Springer 2016; pp. 95-110.
http://dx.doi.org/10.1007/978-3-319-46976-8_11
Raza and Singh
[33] Xu J, Xiang L, Liu Q, et al. mStacked sparse autoencoder (SSAE)
for nuclei detection on breast cancer histopathology images. IEEE
Trans Med Imaging 2016; 35(1): 119-30.
http://dx.doi.org/10.1109/TMI.2015.2458702 PMID: 26208307
[34] Janowczyk A, Basavanhally A, Madabhushi A. Stain Normaliza-
tion using Sparse AutoEncoders (StaNoSA): Application to digital
pathology. Comput Med Imaging Graph 2017; 57: 50-61.
http://dx.doi.org/10.1016/j.compmedimag.2016.05.003 PMID:
27373749
[35] Hatipoglu N, Bilgin G. Cell segmentation in histopathological im-
ages with deep learning algorithms by utilizing spatial relation-
ships. Med Biol Eng Comput 2017; 55(10): 1829-48.
http://dx.doi.org/10.1007/s11517-017-1630-1 PMID: 28247185
[36] Avendi MR, Kheradvar A, Jafarkhani H. Automatic segmentation
of the right ventricle from cardiac MRI using a learning-based ap-
proach. Magn Reson Med 2017; 78(6): 2439-48.
http://dx.doi.org/10.1002/mrm.26631 PMID: 28205298
[37] Su H, Xing F, Kong X, et al. Robust Cell Detection and Segmenta-
tion in Histopathological Images Using Sparse Reconstruction and
Stacked Denoising Autoencoders. Lect Notes Comput Sci 2018;
9351.
[38] Liu S, Liu S, Cai W, et al. Early diagnosis of Alzheimer’s disease
with deep learning. IEEE Int Symp Biomed Imaging 2014;
1015-8.
http://dx.doi.org/10.1109/ISBI.2014.6868045
[39] Cheng J-Z, Ni D, Chou Y-H, et al. Computer-aided diagnosis with
deep learning architecture: applications to breast lesions in US im-
ages and pulmonary nodules in CT scans. Sci Rep 2016; 6: 24454.
http://dx.doi.org/10.1038/srep24454 PMID: 27079888
[40] Miotto R, Li L, Kidd BA, Dudley JT. Deep patient: an unsuper-
vised representation to predict the future of patients from the elec-
tronic health records. Sci Rep 2016; 6: 26094.
http://dx.doi.org/10.1038/srep26094 PMID: 27185194
[41] Cheng LZ. YefengZheng. Deep similarity learning for multimodal
medical images. Comput Methods Biomech Biomed Eng Imaging
Vis 2018; 6(3): 248-52.
http://dx.doi.org/10.1080/21681163.2015.1135299
[42] Huang H, Hu X, Zhao Y, et al. Modeling task fMRI data via deep
convolutional autoencoder. IEEE Trans Med Imaging 2018;
37(7): 1551-61.
http://dx.doi.org/10.1109/TMI.2017.2715285 PMID: 28641247
[43] Hosseini-Asl E, Gimelfarb G, El-Baz A. Alzheimer’s disease diag-
nostics by a deeply supervised adaptable 3D convolutional net-
work. arxiv 2016.
[44] Hou L, Nguyen V, Kanevsky AB, et al. Sparse Autoencoder for
Unsupervised Nucleus Detection and Representation in Histo-
pathology Images. Pattern Recognit 2019; 86: 188-200.
http://dx.doi.org/10.1016/j.patcog.2018.09.007 PMID: 30631215
[45] Hinton G. A practical guide to training restricted boltzmann
machines. Momentum 2010; 9(1): 926.
[46] Yoo Y, Brosch T, Traboulsee A, et al. Deep learning of image fea-
tures from unlabeled data for multiple sclerosis lesion segmenta-
tion. International Workshop on Machine Learning in Medical
Imaging. 117-24.
http://dx.doi.org/10.1007/978-3-319-10581-9_15
[47] Huang H, Hu X, Han J, et al. Latent source mining in FMRI data
via deep neural network. Proceedings of the IEEE Int Symp
Biomed Imaging. 638-41.
http://dx.doi.org/10.1109/ISBI.2016.7493348
[48] Cai Y, Landis M, Laidley DT, Kornecki A, Lum A, Li S. Multi-
modal vertebrae recognition using Transformed Deep Convolu-
tion Network. Comput Med Imaging Graph 2016; 51: 11-9.
http://dx.doi.org/10.1016/j.compmedimag.2016.02.002 PMID:
27104497
[49] Jaumard-Hakoun A, Xu K, Roussel-Ragot P, et al. Tongue con-
tour extraction from ultrasound images based on deep neural net-
work. arxiv 2016.
[50] Cao P, Liu X, Bao H, Yang J, Zhao D. Restricted Boltzmann
machines based oversampling and semi-supervised learning for
false positive reduction in breast CAD. Biomed Mater Eng 2015;
26 (Suppl. 1): S1541-7.
http://dx.doi.org/10.3233/BME-151453 PMID: 26405918
[51] Zhang Q, Xiao Y, Dai W, et al. Deep learning based classification
A Tour of Unsupervised Deep Learning
of breast tumors with shear-wave elastography. Ultrasonics 2016;
72: 150-7.
http://dx.doi.org/10.1016/j.ultras.2016.08.004 PMID: 27529139
[52] van Tulder G, de Bruijne M. Combining generative and discrimi-
native representation learning for lung CT analysis with convolu-
tional restricted boltzmann machines. IEEE Trans Med Imaging
2016; 35(5): 1262-72.
http://dx.doi.org/10.1109/TMI.2016.2526687 PMID: 26886968
[53] Mathews SM, Kambhamettu C, Barner KE. A novel application
of deep learning for single-lead ECG classification. Comput Biol
Med 2018; 99: 53-62.
http://dx.doi.org/10.1016/j.compbiomed.2018.05.013 PMID:
29886261
[54] Pereira S, Meier R, McKinley R, et al. Enhancing interpretability
of automatically extracted machine learning features: application
to a RBM-Random Forest system on brain lesion segmentation.
Med Image Anal 2018; 44: 228-44.
http://dx.doi.org/10.1016/j.media.2017.12.009 PMID: 29289703
[55] Nahid A-A, Mikaelian A, Kong Y. Histopathological breast-im-
age classification with restricted Boltzmann machine along with
backpropagation. Biomed Res (Aligarh) 2018; 29(10): 2068-77.
[56] Bengio Y. Learning Deep Architectures for AI. Found Trends
Mach Learn 2019; 2(1): 1-127.
http://dx.doi.org/10.1561/2200000006
[57] Hinton GE, Dayan P, Frey BJ, Neal RM. The “wake-sleep” algo-
rithm for unsupervised neural networks. Science 1995; 268(5214):
1158-61.
http://dx.doi.org/10.1126/science.7761831 PMID: 7761831
[58] Lee H, Grosse R, Ranganath R, et al. Unsupervised learning of hi-
erarchicalrepresentations with convolutional deep belief networks.
Commun ACM 2011; 54(10): 95-103.
http://dx.doi.org/10.1145/2001269.2001295
[59] Brosch T, Tam R. Manifold learning of brain MRIs by deep learn-
ing. Lect Notes Comput Sci 2013; 8150: 633-40.
http://dx.doi.org/10.1007/978-3-642-40763-5_78
[60] Brosch T, Yoo Y, Li DKB, Traboulsee A, Tam R. Modeling the
variability in brain morphology and lesion distribution in multiple
sclerosis by deep learning. Lect Notes Comput Sci 2014; 8674:
462-9.
http://dx.doi.org/10.1007/978-3-319-10470-6_58
[61] Plis SM, Hjelm DR, Salakhutdinov R, et al. Deep learning for neu-
roimaging: a validation study. Front Neurosci 2014; 8: 229.
http://dx.doi.org/10.3389/fnins.2014.00229 PMID: 25191215
[62] Pinaya WHL, Gadelha A, Doyle OM, et al. Using deep belief net-
work modelling to characterize differences in brain morphometry
in schizophrenia. Sci Rep 2016; 6: 38897.
http://dx.doi.org/10.1038/srep38897 PMID: 27941946
[63] Ortiz A, Munilla J, Górriz JM, Ramírez J. Ensembles of deep
learning architectures for the early diagnosis of the Alzheimer’s
disease. Int J Neural Syst 2016; 26(7)
http://dx.doi.org/10.1142/S0129065716500258 PMID: 27478060
[64] Carneiro G, Nascimento JC, Freitas A. The segmentation of the
left ventricle of the heart from ultrasound data using deep learning
architectures and derivative-based search methods. IEEE Trans Im-
age Process 2012; 21(3): 968-82.
http://dx.doi.org/10.1109/TIP.2011.2169273 PMID: 21947526
[65] Carneiro G, Nascimento JC. Combining multiple dynamic models
and deep learning architectures for tracking the left ventricle endo-
cardium in ultrasound data. IEEE Trans Pattern Anal Mach Intell
2013; 35(11): 2592-607.
http://dx.doi.org/10.1109/TPAMI.2013.96 PMID: 24051722
[66] Ngo TA, Lu Z, Carneiro G. Combining deep learning and level set
for the automated segmentation of the left ventricle of the heart
from cardiac cine magnetic resonance. Med Image Anal 2017; 35:
159-71.
http://dx.doi.org/10.1016/j.media.2016.05.009 PMID: 27423113
[67] Azizi S, Imani F, Ghavidel S, et al. Detection of prostate cancer
using temporal sequences of ultrasound data: a large clinical feasi-
bility study. Int J CARS 2016; 11(6): 947-56.
http://dx.doi.org/10.1007/s11548-016-1395-2 PMID: 27059021
[68] Akhavan Aghdam M, Sharifi A, Pedram MM. Combination of rs-
fMRI and sMRI Data to Discriminate Autism Spectrum Disorders
in Young Children Using Deep Belief Network. J Digit Imaging
Current Medical Imaging, 2021, Vol. 17, No. 00 17
2018; 31(6): 895-903.
http://dx.doi.org/10.1007/s10278-018-0093-8 PMID: 29736781
[69] Salakhutdinov R, Hinton G. Deep Boltzmann machines.Artificial
Intelligence and Statistics. PMLR 2009; pp. 448-55.
[70] Salakhutdinov R, Hinton G. An efficient learning procedure for
deep Boltzmann machines. Neural Comput 2012; 24(8):
1967-2006.
http://dx.doi.org/10.1162/NECO_a_00311 PMID: 22509963
[71] Salakhutdinov R. Learning deep generative models. Annu Rev
Stat Appl 2015; 2: 361-85.
http://dx.doi.org/10.1146/annurev-statistics-010814-020120
[72] Goodfellow I, Mirza M, Courville A, Bengio Y. Multi-prediction
deep Boltzmann machines. Advances in Neural Information Pro-
cessing Systems 2013; 548-56.
[73] Dinggang S, Wu G. SukHeung-Il. Deep Learning in Medical Im-
age Analysis. Annu Rev Biomed Eng 2017; •••: 19.
[74] Suk H-I, Lee S-W, Shen D. Alzheimer’s Disease Neuroimaging
Initiative. Hierarchical feature representation and multimodal fu-
sion with deep learning for AD/MCI diagnosis. Neuroimage 2014;
101: 569-82.
http://dx.doi.org/10.1016/j.neuroimage.2014.06.077 PMID:
25042445
[75] Cao Y, Steffey S, He J, et al. Medical image retrieval: A multimo-
dal approach. Cancer Inform 2015; 13 (Suppl. 3): 125-36.
PMID: 26309389
[76] Wu J, Ruan S, Mazur TR, et al. Heart motion tracking on cine
MRI based on a deep Boltzmann machine-driven level set
method. Proceedings of IEEE Int Symp Biomed Imaging 2018;
1153-6.
http://dx.doi.org/10.1109/ISBI.2018.8363775
[77] Goodfellow JP-A, Mirza M, Xu B, Warde-Farley D. Generative
adversarial nets. Advances in Neural Information Processing Sys-
tems 2014; 2672-80.
[78] Hu Y, Gibson E, Lee L-L, et al. Freehand ultrasound image simu-
lation with spatially-conditioned generative adversarial networks.
Lect Notes Comput Sci 2017; 10555: 105-15.
http://dx.doi.org/10.1007/978-3-319-67564-0_11
[79] Bi L, Kim J, Kumar A, et al. Synthesis of positron emission to-
mography (PET) images via multi-channel generative adversarial
networks (GANs). Lect Notes Comput Sci 2017; 10555: 43-51.
http://dx.doi.org/10.1007/978-3-319-67564-0_5
[80] Bi L, Feng D, Kim J. Dual-Path Adversarial Learning for Fully
Convolutional Network (FCN)-Based Medical Image Segmenta-
tion. Vis Comput 2018; 34(6-8): 1043-52.
http://dx.doi.org/10.1007/s00371-018-1519-5
[81] Iqbal T, Ali H. Generative Adversarial Network for Medical Im-
ages (MI-GAN). J Med Syst 2018; 42(11): 231.
http://dx.doi.org/10.1007/s10916-018-1072-9 PMID: 30315368
[82] Canas K, Liu X, Ubiera B, et al. Scalable biomedical image synth-
esis with GAN. ACM International Conference Proceeding Series.
[83] Jeyaraj P, Nadar ERS. Deep Boltzmann Machine Algorithm for
Accurate Medical Image Analysis for Classification of Cancerous
Region. Cognitive Computation and Systems 2019; 1(3): 85-90.
http://dx.doi.org/10.1049/ccs.2019.0004
[84] Lu N, Li T, Ren X, Miao H. A Deep Learning Scheme for Motor
Imagery Classification based on Restricted Boltzmann Machines.
IEEE Trans Neural Syst Rehabil Eng 2017; 25(6): 566-76.
http://dx.doi.org/10.1109/TNSRE.2016.2601240 PMID:
27542114
[85] Li H, Li X, Ramanathan M, Zhang A. Identifying informative risk
factors and predicting bone disease progression via deep belief net-
works. Methods 2014; 69(3): 257-65.
http://dx.doi.org/10.1016/j.ymeth.2014.06.011 PMID: 24979059
[86] Mardani M, Gong E, Cheng JY, et al. Deep generative adversarial
neural networks for compressive sensing MRI. IEEE Trans Med
Imaging 2019; 38(1): 167-79.
http://dx.doi.org/10.1109/TMI.2018.2858752 PMID: 30040634
[87] Wang Y, Yu B, Wang L, et al. 3D conditional generative adver-
sarial networks for high-quality PET image estimation at low
dose. Neuroimage 2018; 174: 550-62.
http://dx.doi.org/10.1016/j.neuroimage.2018.03.045 PMID:
29571715
[88] Liu Z, Bicer T, Kettimuthu R, Gursoy D, De Carlo F, Foster I. To-
18 Current Medical Imaging, 2021, Vol. 17, No. 00
moGAN: low-dose synchrotron x-ray tomography with generative
adversarial networks: discussion. J Opt Soc Am A Opt Image Sci
Vis 2020; 37(3): 422-34.
http://dx.doi.org/10.1364/JOSAA.375595 PMID: 32118926
[89] Kang E, Koo HJ, Yang DH, Seo JB, Ye JC. Cycle-consistent ad-
versarial denoising network for multiphase coronary CT angiogra-
phy. Med Phys 2019; 46(2): 550-62.
http://dx.doi.org/10.1002/mp.13284 PMID: 30449055
[90] Frid-Adar M, Diamant I, Klang E, et al. GAN-based synthetic
medical image augmentation for increased CNN performance in
liver lesion classification. Neurocomputing 2018; 321(10):
321-31.
http://dx.doi.org/10.1016/j.neucom.2018.09.013
[91] Chuquicusma MJM, Hussein S, Burt J, et al. How to fool radiolo-
gists with generative adversarial networks?A visual turing test for
lung cancer diagnosis. Proceedings - IEEE Int Symp Biomed
Imaging 2018; 240-4.
[92] Mondal AK, Dolz J, Desrosiers C. Few-shot 3D Multi-modal Med-
ical Image Segmentation using Generative Adversarial Learning.
arXiv preprint 2018.
[93] Salehinejad H, Valaee S, Dowdell T, et al. Generalization of Deep
Neural Networks for Chest Pathology Classification in X-Rays Us-
ing Generative Adversarial Networks. Proceeding IEEE Internatio-
nal Conference on Acoustics, Speech and Signal Processing.
990-4.
http://dx.doi.org/10.1109/ICASSP.2018.8461430
[94] Madani A, Moradi M, Karargyris A, et al.
[95] Baur C, Albarqouni S, Navab N. MelanoGANs : High Resolution
Skin Lesion Synthesis with GANs. arXiv preprint 2018.
[96] Lahiri A, Jain V, Mondal A, et al. Retinal Vessel Segmentation
Under Extreme Low Annotation: A Gan Based Semi-Supervised
Approach. IEEE International Conference on Image Processing (I-
CIP). 418-22.
http://dx.doi.org/10.1109/ICIP40778.2020.9190882
[97] Costa P, Galdran A, Meyer MI, et al. End-to-End Adversarial Reti-
nal Image Synthesis. IEEE Trans Med Imaging 2018; 37(3):
781-91.
http://dx.doi.org/10.1109/TMI.2017.2759102 PMID: 28981409
[98] Zhao H, Li H, Maurer-Stroh S, Cheng L. Synthesizing retinal and
neuronal images with generative adversarial nets. Med Image
Anal 2018; 49: 14-26.
http://dx.doi.org/10.1016/j.media.2018.07.001 PMID: 30007254
[99] Shin HC, Tenenholtz NA, Rogers JK, et al. Medical image synthe-
sis for data augmentation and anonymization using generative ad-
versarial networks. Lect Notes Comput Sci 2018; •••: 1-11.
http://dx.doi.org/10.1007/978-3-030-00536-8_1
[100] Mok TCW, Chung ACS. Learning data augmentation for brain tu-
mor segmentation with coarse-to-fine generative adversarial net-
works. Lect Notes Comput Sci 2019; 11383.
http://dx.doi.org/10.1007/978-3-030-11723-8_7
[101] Tom F, Sheet D. Simulating patho-realistic ultrasound images us-
ing deep generative networks with adversarial learning. Proceed-
ings IEEE Int Symp Biomed Imaging. Washington DC. 2018; pp.
1174-7.
http://dx.doi.org/10.1109/ISBI.2018.8363780
[102] Singh NK, Raza K. Medical Image Generation using Generative
Adversarial Networks. Stud Comput Intell
[103] Ghoneim A, Muhammad G, Amin SU, et al. Medical Image
Forgery Detection for Smart Healthcare. IEEE Commun Mag
2018; 56(4): 33-7.
http://dx.doi.org/10.1109/MCOM.2018.1700817
[104] Zhu Q. Exploiting interslice correlation for MRI prostate image
segmentation, from recursive neural networks aspect. Complexity
2018; (Feb): 2018.
http://dx.doi.org/10.1155/2018/4185279
[105] Golea NE-H, Melkemi KE. ROI-based fragile watermarking for
medical image tamper detection. International Journal of High-Per-
formance Computing and Networking 2019; 13(2): 199-210.
http://dx.doi.org/10.1504/IJHPCN.2019.097508
[106] Dorgham O, Al-Rahamneh B, Ai-Hadidi M, Khatatneh KF, Almo-
mani A. Enhancing the security of exchanging and storing DI-
COM medical images on the cloud. Int J Cloud Appl Comput
2018; 8(1): •••.
Raza and Singh
http://dx.doi.org/10.4018/IJCAC.2018010108
[107] Guo P, Evans A, Bhattacharya P. Nuclei segmentation for quantifi-
cation of brain tumors in digital pathology images. Int J Softw Sci
Comput Intell 2018; 10(2): •••.
http://dx.doi.org/10.4018/IJSSCI.2018040103
[108] Liu H, Guo Q, Wang G, Gupta BB, Zhang C. Medical image reso-
lution enhancement for healthcare using nonlocal self-similarity
and low-rank prior. Multimedia Tools Appl 2019; 78(7)
http://dx.doi.org/10.1007/s11042-017-5277-6
[109] Zhu Q, Du B, Turkbey B, et al. Deeply-supervised CNN for pros-
tate segmentation 2017 International Joint Conference on Neural
Networks. 178-84.
http://dx.doi.org/10.1109/IJCNN.2017.7965852
[110] Zhu Q, Du B, Yan P. Boundary-weighted domain adaptive neural
network for prostate MR image segmentation. IEEE Trans Med
Imaging 2020; 39(3): 753-63.
http://dx.doi.org/10.1109/TMI.2019.2935018 PMID: 31425022
[111] Sital C, Brosch T, Tio D, Raaijmakers A, Weese J. 3D medical im-
age segmentation with labeled and unlabeled data using autoen-
coders at the example of liver segmentation in CT images. arXiv
preprint 2020.
[112] Larrazabal AJ, Martínez C, Glocker B, Ferrante E. Post-dae: Ana-
tomically plausible segmentation via post-processing with denois-
ing autoencoders. IEEE Trans Med Imaging 2020; 39(12):
3813-20.
http://dx.doi.org/10.1109/TMI.2020.3005297 PMID: 32746125
[113] Kazlouski S. Tuberculosis CT Image Analysis Using Image Fea-
tures Extracted by 3D Autoencoder. International Conference of
the Cross-Language Evaluation Forum for European Languages.
131-40.
http://dx.doi.org/10.1007/978-3-030-58219-7_12
[114] Ilse M, Tomczak JM, Louizos C, Welling M. Domain invariant
variational autoencoders.Medical Imaging with Deep Learning
2020; 322-48.
[115] Amin J, Sharif M, Gul N, et al. Brain Tumor Detection by Using
Stacked Autoencoders in Deep Learning. J Med Syst 2019; 44(2):
32.
http://dx.doi.org/10.1007/s10916-019-1483-2 PMID: 31848728
[116] Mendoza-Léon R, Puentes J, Uriza LF, Hernández Hoyos M. Sin-
gle-slice Alzheimer’s disease classification and disease regional
analysis with Supervised Switching Autoencoders. Comput Biol
Med 2020; 116
http://dx.doi.org/10.1016/j.compbiomed.2019.103527 PMID:
31765915
[117] Spatiotemporal Attention Autoencoder (STAAE) for ADHD Clas-
sification Dong Q, Qiang N, Lv J, Li X, Liu T, Li Q. Spatiotempo-
ral Attention Autoencoder (STAAE) for ADHD Classifica-
tion.Lect. Notes Comput. Sci2020; 12267.
http://dx.doi.org/10.1007/978-3-030-59728-3_50
[118] Hecht H, Sarhan MH, Popovici V. Disentangled Autoencoder for
Cross-Stain Feature Extraction in Pathology Image Analysis. Appl
Sci (Basel) 2020; 10(18): 6427.
http://dx.doi.org/10.3390/app10186427
[119] Dong Q, Qiang N, Lv J, et al. Discovering Functional Brain Net-
works with 3D Residual Autoencoder (ResAE). Lect Notes Com-
put Sci 2020; 12267.
http://dx.doi.org/10.1007/978-3-030-59728-3_49
[120] Adarsh R, Amarnageswarao G, Pandeeswari R, Deivalakshmi S.
Dense Residual Convolutional Auto Encoder For Retinal Blood
Vessels Segmentation. 2020 6th International Conference on Ad-
vanced Computing and Communication Systems (ICACCS).
280-4.
http://dx.doi.org/10.1109/ICACCS48705.2020.9074172
[121] Li D, Fu Z, Xu J. Stacked-autoencoder-based model for
COVID-19 diagnosis on CT images. Appl Intell 2020.
http://dx.doi.org/10.1007/s10489-020-02002-w
[122] Reddy AVN, Krishna CP, Mallick PK, et al. Analyzing MRI
scans to detect glioblastoma tumor using hybrid deep belief net-
works. J Big Data 2020; 7: 35.
http://dx.doi.org/10.1186/s40537-020-00311-y
[123] Gopal A, Gandhimaruthian L, Ali J. Role of General Adversarial
Networks in Mammogram Analysis: A Review. Curr Med Imag-
ing 2020; 16(7): 863-77.
A Tour of Unsupervised Deep Learning
http://dx.doi.org/10.2174/1573405614666191115102318 PMID:
33059556
[124] Wolterink JM, Kamnitsas K, Ledig C, Išgum I. Generative adver-
sarial networks and adversarial methods in biomedical image anal-
ysis. arXiv preprint 2018.
[125] Yi X, Walia E, Babyn P. Generative adversarial network in medi-
cal imaging: A review. Med Image Anal 2019; 58
http://dx.doi.org/10.1016/j.media.2019.101552 PMID: 31521965
[126] Kazeminia S, Baur C, Kuijper A, et al. GANs for medical image
analysis. Artif Intell Med 2020; 9
http://dx.doi.org/10.1016/j.artmed.2020.101938
Current Medical Imaging, 2021, Vol. 17, No. 00 19
[127] Zhang Y, Miao S, Mansi T, Liao R. Unsupervised X-ray image
segmentation with task driven generative adversarial networks.
Med Image Anal 2020; 62
http://dx.doi.org/10.1016/j.media.2020.101664 PMID: 32120268
[128] Rezaei M, Yang H, Meinel C. Recurrent generative adversarial
network for learning imbalanced medical image semantic segmen-
tation. Multimedia Tools Appl 2020; 79(21): 15329-48.
http://dx.doi.org/10.1007/s11042-019-7305-1
[129] Lei B, Xia Z, Jiang F, et al. Skin lesion segmentation via genera-
tive adversarial networks with dual discriminators. Med Image
Anal 2020; 64
http://dx.doi.org/10.1016/j.media.2020.101716 PMID: 32492581