See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/353567135

Addressing data imbalance problems in ligand-binding site prediction using a

variational autoencoder and a convolutional neural network

Article  in  Briefings in Bioinformatics · July 2021

DOI: 10.1093/bib/bbab277

CITATIONS READS

0 51

3 authors:

Trung-Duong Nguyen-Trinh Duc-Khanh Nguyen

Novo Nordisk Protein Research Center, University of Copenhagen Yuan Ze University
19 PUBLICATIONS   136 CITATIONS    4 PUBLICATIONS   2 CITATIONS   

SEE PROFILE SEE PROFILE

Yu Yen Ou
Yuan Ze University
67 PUBLICATIONS   1,279 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

bioinformatics View project

Using NLP approach on identifying molecular functions of a variety of proteins View project

All content following this page was uploaded by Trung-Duong Nguyen-Trinh on 01 August 2021.

The user has requested enhancement of the downloaded file.

 Briefings in Bioinformatics, 00(00), 2021, 1–10

https://doi.org/10.1093/bib/bbab277
Problem Solving Protocol

Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021

Addressing data imbalance problems in
ligand-binding site prediction using a variational
autoencoder and a convolutional neural network
Trinh-Trung-Duong Nguyen, Duc-Khanh Nguyen and Yu-Yen Ou
Corresponding author: Yu-Yen Ou, Department of Computer Science and Engineering, Yuan Ze University, Chung-Li 32003, Taiwan.
Tel.: +886-3-4638800 #2185; E-mail: yien@saturn.yzu.edu.tw

Abstract
Since 2015, a fast growing number of deep learning–based methods have been proposed for protein–ligand binding site
prediction and many have achieved promising performance. These methods, however, neglect the imbalanced nature of
binding site prediction problems. Traditional data-based approaches for handling data imbalance employ linear
interpolation of minority class samples. Such approaches may not be fully exploited by deep neural networks on
downstream tasks. We present a novel technique for balancing input classes by developing a deep neural network–based
variational autoencoder (VAE) that aims to learn important attributes of the minority classes concerning nonlinear
combinations. After learning, the trained VAE was used to generate new minority class samples that were later added to the
original data to create a balanced dataset. Finally, a convolutional neural network was used for classification, for which we
assumed that the nonlinearity could be fully integrated. As a case study, we applied our method to the identification of FAD-
and FMN-binding sites of electron transport proteins. Compared with the best classifiers that use traditional machine
learning algorithms, our models obtained a great improvement on sensitivity while maintaining similar or higher levels of
accuracy and specificity. We also demonstrate that our method is better than other data imbalance handling techniques,
such as SMOTE, ADASYN, and class weight adjustment. Additionally, our models also outperform existing predictors in
predicting the same binding types. Our method is general and can be applied to other data types for prediction problems
with moderate-to-heavy data imbalances.

Key words: variational autoencoder; convolutional neural network; protein–ligand binding site prediction; data imbalance
handlin; gelectron transport proteins

Introduction activities. In the areas of molecular docking, pharmaceutical

Proteins perform their biological functions through interactions interaction, compound design, prediction of ligand affinity,
with other molecules. Accurately locating interaction sites or and even molecular dynamics, LBSs have received considerable
ligand-binding sites (LBSs) on proteins is therefore an initial step attention. Identifying LBSs not only helps explore intermolec-
in understanding the biological mechanisms underlying protein ular mechanisms, but it also effectively helps explain the

Trinh-Trung-Duong Nguyen is now a postdoctoral researcher at Computer science department of Yuan Ze University, Taiwan. She is interested in
application of machine (deep) learning in protein bioinformatics.
Duc-Khanh Nguyen received M.S Degree in Department of Information Management, Yuan Ze University, Taiwan. He is now working toward the Ph.D
degree in the Department of Information Management, Yuan Ze university. His current research interest include applying Artificial Intelligent, Machine
learning, Deep learning in smart manufacturing and healthcare.
Yu-Yen Ou is an Associate Professor in the Department of Computer Science and Engineering, Graduate Program in Biomedical Informatics, Yuan Ze
University, Taiwan. He received the B.S. degree in Department of Math and Computer Science Education, Taipei Municipal Teachers College and the
Ph.D. degree in Department of Computer Science and Information Engineering, National Taiwan University, Taiwan. His fields of professional interest
are Bioinformatics, Machine Learning, and Data Mining.
Submitted: 31 March 2021; Received (in revised form): 29 June 2021

© The Author(s) 2021. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com
1
 2 Nguyen et al.
pathogenesis of diseases, providing insights into drug discovery
and design. However, in the majority of cases, experimental
details related to protein–ligand interactions are lacking. The
latest version (5 February 2021) of the BioLiP database [1] shows
that only 299 051 of over 529 047 (56.5%) collected entries have
regular ligand information, which means that addressing the
ligand-binding sites problem is an important and urgent task for
further related work. Given the time-consuming and inefficient
process of biochemical experiments in the identification of LBSs
on proteins, an effective computational approach is essential.
Over the last 5 years, deep learning has demonstrated its
power in protein–ligand binding prediction problems due to its
outstanding performance compared to conventional machine
learning methods. Many groups have exploited deep neural net-
works, either in a single architecture, or in hybrid architectures
and ensemble approaches, to extract discriminative features for
classifying residues of interest into binding or nonbinding types.
The ligand types of interest are metals (Ca2+ , Fe3+ , Mg2+ , Mn2+ ,
Na+ , Zn2+ , etc.), biologically relevant molecule bindings (ADP,
ATP, FMN, FAD, GTP, Heme, NAD, PO3− 2−
4 , SO4 , etc.) and nucleic
acids. In Supplementary Table S1, available online at http://bib.o
xfordjournals.org/, of the Supplementary data, we list 18 studies
published from 2016 to the beginning of 2021 that use deep
learning–based models to predict protein–ligand binding sites.
We can see from the list that only 6 out of 18 studies (one-third)
employed at least one data imbalance handling technique.
Binary classification on imbalanced datasets usually suffers
from biased decisions in which most standard classification
algorithms favor the majority class, leading to poor accuracy in
minority class prediction. Data imbalance handling techniques
mainly fall into either of two categories: a data-based approach
or an algorithm-based approach [2]. The data-based, or sam-
pling, approach, which uses the techniques of undersampling
the majority class or oversampling the minority class, is more
common. As the first technique discards important informa-
tion of the majority class, the second technique may be the
better choice. With oversampling techniques, if new positive
samples are created merely by being copied from samples of the
minority class, high false-positive rates may be seen. Improved
techniques for generating new positive samples require creat-
ing synthetic samples from minority class samples that add
more meaningful information than does the duplication process.
SMOTE [3], ADASYN [4], and BorderlineSMOTE [5] are repre-
sentatives of this approach. In biomedical research, which has
a broader scope than ligand-binding site prediction, SMOTE-
based and ADASYN-based algorithms have been widely applied
in dealing with imbalanced data [6–18]. Despite the popularity
of these techniques, it is important to note that the generated
synthetic samples are linear combinations of the features of
the minority class samples. However, it is well-known that one
power of deep neural networks is their ability to learn from
nonlinearly separable data. Given the fact that most real-world
datasets are nonlinearly separable, we assumed that if new
samples were generated with the addition of nonlinearity, the
downstream tasks, (i.e. binary classification using deep neural
networks) would be able to fully integrate this nonlinearity and
thus learn optimally from the generated data. Based on the above
idea, we propose a new approach using a variational autoen-
coder (VAE) to produce minority class samples. To generate new
samples and balance the datasets, positive samples representing
the input are passed through the encoder and mapped to a
continuous latent space, Z, and then a latent representation
sample from Z is decoded by a decoder. The whole process is
optimized for two tasks: reconstructing the input from Z and
learning the probability distribution of the input. As members
of a diverse class of generative deep learning architectures, VAE-
based generative models have achieved good results generating
images [19], speech [20], sentences [21] and de novo molecular
designs [22], to name a few. With the VAE, we hoped to generate
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
new high-quality minority class samples for classification. For
the prediction tasks, we employed deep convolutional neural
networks to optimize prediction performance.
As a case study, we applied our method to the identification
of flavin adenine dinucleotide (FAD) and flavin mononucleotide
(FMN) binding sites of electron transport proteins. In electron
transport chains (ETCs) composed of four protein complexes (I,
II, III and IV) and adenosine triphosphate (ATP) synthase, FMN
is one part of Complex I, while FAD is involved in Complex
II activities (Figure 1). In both the Krebs cycle and oxidative
phosphorylation, FAD acts as an electron carrier. It accepts and
transfers electrons to FADH2. Then, FADH2 transfers the com-
pound to Complex II. Several ATP molecules are generated for
each pair of FADH2 passing through the ETC. FAD also affects
enzymes in charge of synthesizing other vital coenzymes such
as Nicotinamide adenine dinucleotide. Serious riboflavin defi-
ciency may result in insufficient coenzyme levels, poor energy
metabolism and consequent energy depletion. Abnormal energy
metabolism is known to be associated with many diseases,
including Huntington’s disease [23], diabetes [24], neuromuscu-
lar and neurological disorders and cancer [25]. Moreover, flavo-
protein deficiencies occurring in metabolic pathways can cause
human disorders such as glutaric acidemia and Leigh syndrome
[26]. The identification of the FMN group in Complex I provides
a pharmacologically accessible target for delaying aging and
treating neurodegenerative diseases such as Parkinson’s [27].
The improvement in mitochondrial respiration with FAD sup-
plementation may reduce frataxin deficiency [28]. In the drug
design industry, FMN riboswitch is an emerging target for the
development of novel RNA-targeting antibiotics [29], while the
generation of FAD analogs is a useful target for inhibiting bac-
terial infection [30]. With these important roles and the effects
of FAD- and FMN-binding functions in electron transport pro-
teins, an efficient computational approach for predicting them
is essential for biologists and other researchers.
Our contributions are as follows: First, we propose a novel
approach based on a deep neural network for handling data
imbalances. To the best of our knowledge, we are the first group
to apply a variational autoencoder to this long-standing prob-
lem. Although we carried out our experiments for the prediction
of FAD- and FMN-binding sites of electron transport proteins,
this method is general and can be applied to other types of
data. Second, we analyzed the effect of varying sample vector
dimensions in latent space Z and the effect of different distri-
butions of both reconstruction and latent losses on the final
performance. Third, we benchmarked our proposed approach
with conventional techniques for handling imbalanced data
(SMOTE, BorderlineSMOTE, ADASYN and class weight adjust-
ment). Finally, we analyzed the effectiveness of our prediction
models by comparing them with traditional machine learning
algorithms and existing studies on the same binding types.
Materials and methods
Our workflow, depicted in Figure 2, is as follows: (i) First, the raw
protein sequences are converted into position-specific scoring
matrices (PSSMs) of size 15 × 20 (part A). (ii) Subsequently, only
the PSSMs of the positive training data samples are flattened
to put into the variational autoencoder. The end result of this
process is newly generated positive samples (part B). (iii) Next,
the generated positive samples, along with the original intact

Figure 1. The roles of FAD and FMN in an electron transport chain.
negative training samples, are input into a convolutional neural
network for training the prediction models (part C). Finally, the
trained models from Step 3 are used for predicting with unseen
samples as an independent test (part D).
Benchmark dataset
We collected raw sequences of electron transport proteins from
the Universal Protein Resource (UniProt) [31] database (release
2021-01). To create high-quality, unbiased datasets, we adopted
rigorous criteria for selecting data, as follows: (i) The query was
structured so that the resulting sequences have been reviewed
and were not fragmented. After this exclusion, 475 FAD- and
466 FMN-binding electron transport protein sequences were
retrieved. (ii) Proteins lacking FAD- and FMN-binding sites were
also excluded, which reduced the number of FAD- and FMN-
binding sequences to 228 and 105, respectively. (iii) PSI Blast [32]
was applied to discard sequences with a similarity level greater
than 20%. After this step, 32 FAD- and 11 FMN-binding sequences
remained and were used in this study. (iv) Each dataset was
divided into two parts: a cross-validation portion (26 FAD- and
8 FMN-binding sequences) for model construction and an inde-
pendent test portion (6 FAD- and 3 FMN-binding sequences) for
model evaluation. (v) The description files with binding positions
were used to determine bound and unbound residues with only
the annotations determined by expert’s experiments. (Evidence
code ECO 0000250 was not applied.) Supplementary Table S2,
available online at http://bib.oxfordjournals.org/, of the Supple-
mentary data displays the number of binding sites in each part.
Representations of FAD- and FMN-binding electron
transport proteins
PSSM is a matrix containing positive and negative integers
that represent motifs in biological sequences. Large positive
scores often indicate functional residues such as active site
residues or internal interaction sites. Therefore, in binding site
Addressing data imbalance using deep learning 3
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
prediction problems, PSSM has been an indispensable part of
feature extraction processes. Chen et al. [33, 34] discovered that
excluding PSSM profiles resulted in a greater decrease in
prediction performance than including other feature types.
We used a nonredundant protein database and PSI-BLAST
[32] software with three iterations to convert raw sequences into
PSSMs. Then, for each PSSM profile, we slid a window of size S
along the matrix to generate N smaller matrices of size n × 20
with N being the length of the PSSM. If the center position is
a binding residue, the corresponding matrix is regarded as a
positive sample and as a negative sample otherwise. In this way,
the prediction for the central amino acid is supported by the
information of the surrounding amino acids. Many groups often
perform a preliminary experiment to find the best value for S
among common choices of odd numbers, i.e. from 15 to 23. We
adopted this approach using a quick and simple Random Forest
classification algorithm with computationally efficient window
sizes from 11 to 19. We found that the area under the receiver
operating characteristic curve performance on both datasets
was highest with a window size of 15. (The performance details
are presented in Supplementary Figure S1, available online at
http://bib.oxfordjournals.org/, of the Supplementary data). We
accordingly chose a window size of 15 for further experiments.
Imbalanced data handling with a variational
autoencoder
An autoencoder is a self-supervised neural network that learns
to encode the input x into a low-dimensional vector z, then
decodes and reconstructs the data so that the output d(z) is as
close to the input as possible. Dimension reduction for visual-
ization and noise reduction are common applications of autoen-
coders. In an autoencoder, the cost of the task is the reconstruc-
tion loss and the low-dimensional encoded values are referred
to as the latent representation or bottleneck.
Extending the idea of the autoencoder, a VAE regularizes
the encoding distribution during training so that the latent
 4 Nguyen et al.
Figure 2. The flowchart of our study.
space has good properties for enabling new data generation. By
sampling different points from the latent space and decoding
them, many new data points bearing similar characteristics to
the input can be created for use in downstream tasks. In order
to generate new high-quality data points, a constraint is imposed
on learning the latent space so that it stores the latent attribute
as a probability distribution. Therefore, the loss of this task
consists of two parts: the reconstruction loss, which is similar to
that of the autoencoder, and the latent loss. Figure 3 describes
the information flow of the two architectures for comparison.
We constructed a VAE with encoder and decoder con-
volutional neural networks to generate new samples for
minority classes (FAD- and FMN-binding sites). Part B of Figure 2
illustrates this process. The representation based on a position-
specific scoring matrix input was flattened to a vector and
then inputted into the encoder. We assumed that the latent
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
distribution is a Gaussian distribution, and we trained the
VAE to learn the mean and covariance of the distribution
jointly with the goal of reconstructing the output with minimal
reconstruction errors. After training, our encoder outputted
the means and covariances from which we sampled new
latent vectors for generating new samples by passing them
through the decoder. The number of generated samples for
each binding site prediction problem was calculated so that
we obtained a balanced input dataset for training. We kept the
independent datasets intact, which means that they remained
imbalanced.
Variational autoencoder structure
The structure of the convolutional autoencoder contains the
basic building blocks of convolutional neural networks, such as
 Addressing data imbalance using deep learning 5

Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021

Figure 3. The information flow of a variational autoencoder compared to an autoencoder.

a convolutional layer, a max pooling layer (or upsampling layer), Evaluation metrics
a batch normalization layer and an activation layer. The input
Prediction performance was evaluated by common standard
layer is a vector of size 15 × 20 that connects to the first 1D
metrics such as accuracy, specificity, sensitivity, Matthews cor-
convolutional layer of the encoder and is followed by a batch
relation coefficient (MCC) and the area under the receiver oper-
normalization layer and a max pooling layer. We used the Leaky
ating characteristic curve. In addition, we used the geometric
Rectified Linear Unit (LeakyReLu) activation function in all layers
mean (G-mean) of the accuracy rates calculated separately for
except the final layer. We also carried out the experiment with
the majority and minority classes as a single metric for compar-
different numbers of intermediate convolutional layers having
ison and for tuning the models. The G-mean has been used to
1024, 512 and 256 nodes interspersed with batch normalization
evaluate performance of learning algorithms with imbalanced
layers and max pooling layers. The output of the encoder is
datasets in a lot of studies [36–43]. Formulas for all the metrics
an m-dimensional latent layer with m being a hyperparameter.
can be seen in Supplementary Figure S2, available online at
The decoder was designed to be symmetric with the encoder. In
http://bib.oxfordjournals.org/, of the Supplementary data.
training our VAE, the loss function was composed of a recon-
struction term, which makes the encoding–decoding scheme
efficient, and a regularization term, which makes the latent
Results and discussion
space regular. The first term is specified as the mean squared
error and the second term is the Kullback–Leibler divergence [35], Visual representation of positive samples generated
which measures the difference between two distributions. The from the VAEs
loss function is then formulated as
To understand the distribution of generated samples from the
VAEs in comparison with the distribution of original data, we
used t-SNE, a popular algorithm for dimension reduction, to
VAEloss =|| x − d(z)||2 + KL [N (μx , σx ) , N (0, 1)] (1)
transform samples of 300 dimensions into samples of 2 dimen-
sions and plotted them in a two-dimensional map. Supplemen-
tary Figures S3 and S4, available online at http://bib.oxfordjou
where N(μx, σ x) and N(0,1) are the learning latent distribution
rnals.org/, in the Supplementary data present the projection
and standard normal distribution, respectively.
of original positive samples and generated positive samples
from VAE for FAD- and FMN-binding datasets, respectively. We
Convolutional neural network structure can observe from the figures that the distribution of generated
positive samples is similar to those of real positive samples.
Our 1D convolutional neural networks (1DCNN) serve as
This means that the generated samples bear the characteristics
the classifiers with the input being the balanced datasets
of the original data and can be used efficiently to balance the
during the training processes. In general, the 1DCNN scheme
datasets.
is Input-(1DConvolution-Dropout-MaxPooling-) × k-Flatten-
FullyConnected-Dropout-FullyConnected-Softmax with k being
learnable and less than 4. We also used the LeakyReLu
activation function in all layers except the final layer. The
Hyperparameter tuning processes
Softmax layer contains two nodes for binding and nonbinding As our process involves two different deep neural networks, a
classes. In training the CNN classifiers, we used the binary VAE for generating new samples and a CNN for classification,
cross-entropy loss. much effort was put into choosing and tuning hyperparameters
 6 Nguyen et al.
Figure 4. (A) G-mean scores on an FAD-binding independent test over different latent vector dimensions. (B) G-mean scores on an FMN-binding independent test over
different latent vector dimensions.
of the two networks. However, both networks employed convolu-
tional neural networks as their components. Basically, we used
the training data for training and tuning the hyperparameters
and the independent test data for evaluating and comparing
the optimal learned models. We tried different architectures
(described above), and a grid search over the hyperparameters
was performed for each architecture. For the VAEs, we con-
sider three hyperparameters, namely, the number of epochs, the
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
number of batch sizes and the learning rate. For the CNNs, apart
from the hyperparameters used in VAEs, we also considered
dropout values and weight decays. The optimization algorithm
for both neural networks was fixed with Adam optimizer [44].
Moreover, we present in the Supplementary data the search
ranges for all hyperparameters, the best hyperparameter sets
(Supplementary Tables S3 and S4 available online at http://bi
b.oxfordjournals.org/), the best architectures for the variational

Figure 5. G-mean scores on FAD- and FMN-binding independent tests over different distributions of reconstruction loss and latent loss.
autoencoder and the 1DCNN-based classifier (Supplementary
Tables S5 and S6 available online at http://bib.oxfordjournals.o
rg/).
Effectiveness of varying sample sizes in the latent space
After obtaining the optimal models, we were interested in
analyzing the effect of varying vector dimensions of the samples
in Z on the prediction of unseen samples using independent
datasets. We constructed the VAE with various dimensions,
denoted as dim(z), from 2 to 64. For each dimension, we repeated
the prediction experiments 10 times and calculated the average,
maximum, minimum and standard deviation of the G-mean
performance. (Please see Figure 4A and B for FAD- and FMN-
binding site predictions on independent datasets, respectively.)
From these figures, we observe that the FAD-binding site
prediction model achieves the highest maximum and highest
average G-mean when dim(z) is equal to 16. Regarding FMN-
binding site prediction, the models with dim(z) = 16 also obtained
the highest average score and lowest standard deviation score.
Considering the robustness of the models, we chose the models
with the highest average G-means (dim(z) = 16 in both cases) for
the next analysis.
Effectiveness of various contributions of reconstruction
loss and latent loss
In this analysis, we varied the contribution of the reconstruction
loss and the latent loss to the total loss of the VAE models. We
specified the total loss of the VAE as
Total loss = a ∗ reconstruction_loss + (1 − a) ∗ latent_loss
with α chosen from a list of 0.1, 0.2, 0.3, 0.4 and 0.5.
The best models from the previous analysis were chosen
for this analysis on independent datasets. We report the best
G-mean scores over five experiments for each value of α in
Figure 5, where it can be seen that the distributions of recon-
struction loss and latent loss have an impact on the prediction
performance. The G-mean scores range from 0.72 to 0.85 for
FAD-binding site prediction and from 0.67 to 0.71 for FMN-
binding site prediction. However, for both datasets, the trends are
not clear.
Addressing data imbalance using deep learning 7
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
Performance comparison with other data imbalance
handling techniques
In this section, we report the results of our analysis on the
effectiveness of using a variational autoencoder for handling
data imbalances. We employed a variety of data imbalance han-
dling algorithms, such as SMOTE, BorderlineSMOTE, ADASYN
and class weight adjustment, to generate balanced datasets and
classified them with the convolutional neural networks. We
also carried out experiments where no data imbalance handling
technique was used. We also searched for the optimal hyperpa-
rameters for the convolutional neural networks. To make our
analysis results more robust and reliable, we repeated each
experiment 10 times and averaged the results. Table 1 displays
the performance comparison for the independent test, with the
best sensitivity and MCC values highlighted in bold. All standard
deviation (STD) values are also presented.
From Table 1, we observe that the combination of VAE and
CNN yields the best G-mean, sensitivity and MCC performance
on both the FAD and FMN independent tests. For the FAD dataset,
our model obtained the best G-mean (0.72), which is far bet-
ter than the second-best technique (CNN + Classweight with G-
mean = 0.64). For the FMN dataset, our method is also better than
the best approach (CNN + BorderlineSMOTE with G-mean = 0.68).
The reason for less improvement may be the smaller number
of samples in the FMN training dataset compared to the FAD
training dataset.
Performance comparison with traditional machine
learning algorithms
We further compared the effectiveness of our method with
(2)
several traditional machine learning algorithms, namely, Ran-
domForest, Support Vector Machine, XGBoost and AdaBoost (pre-
viously called ADaBoost.M1). For each of these algorithms, we
searched for the best hyperparameters as well as best data
imbalance handling algorithms using a grid search and recorded
the best G-mean performance. Similar to experiments reported
in Table 1, we repeated each experiment 10 times and averaged
the best results. Table 2 shows the independent test perfor-
mance on five standard metrics for FAD and FMN datasets, with
the highest scores highlighted in bold. Our approach, with G-
mean values of 0.76 and 0.73, MCC values of 0.48 and 0.51 on
FAD and FMN independent test data, respectively, outperforms
 8 Nguyen et al.

Table 1. Performance comparison among different data imbalance handling techniques on independent tests. (Each experiment was repeated
10 times and average results are reported in format: m ± d, where m is the average and d is the standard deviation across the 10 experiments)

FAD

Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021

Model Acc Spec Sen MCC G-mean

CNN+ Adasyn 0.98 ± 0.0 1.00 ± 0.0 0.40 ± 0.09 0.52 ± 0.04 0.62 ± 0.07
CNN+ Smote 0.98 ± 0.0 1.00 ± 0.0 0.36 ± 0.07 0.49 ± 0.03 0.60 ± 0.06
CNN+ BordelineSmote 0.98 ± 0.0 1.00 ± 0.0 0.40 ± 0.06 0.53 ± 0.04 0.63 ± 0.05
CNN+ Classweight 0.95 ± 0.01 0.96 ± 0.01 0.44 ± 0.11 0.26 ± 0.07 0.64 ± 0.08
CNN+ NoBalance 0.98 ± 0.0 1.00 ± 0.0 0.37 ± 0.08 0.53 ± 0.06 0.61 ± 0.06
CNN+ VAE (Proposed model) 0.98 ± 0.01 0.99 ± 0.01 0.52 ± 0.01 0.48 ± 0.05 0.72 ± 0.07
FMN

Acc Spec Sen MCC G-mean

CNN+ Adasyn 0.88 ± 0.01 0.95 ± 0.01 0.48 ± 0.06 0.47 ± 0.04 0.67 ± 0.04
CNN+ Smote 0.88 ± 0.01 0.95 ± 0.01 0.48 ± 0.06 0.47 ± 0.04 0.67 ± 0.04
CNN+ BordelineSmote 0.88 ± 0.01 0.94 ± 0.01 0.50 ± 0.08 0.48 ± 0.06 0.68 ± 0.06
CNN+ Classweight 0.84 ± 0.02 0.93 ± 0.03 0.31 ± 0.09 0.27 ± 0.04 0.53 ± 0.07
CNN+ NoBalance 0.86 ± 0.01 0.95 ± 0.01 0.28 ± 0.05 0.29 ± 0.04 0.51 ± 0.04
CNN+ VAE (Proposed model) 0.88 ± 0.01 0.94 ± 0.01 0.52 ± 0.05 0.49 ± 0.04 0.70 ± 0.03

Table 2. Comparison between the proposed method and traditional machine learning algorithms for FAD- and FMN-binding site prediction (Each
experiment was repeated 10 times and average results are reported in format: m ± d, where m is the average and d is the standard deviation
across the 10 experiments)

FAD

Predictor Acc Spec Sen MCC G-mean

Random Forest 0.98 ± 0.0 0.99 ± 0.0 0.44 ± 0.0 0.43 ± 0.0 0.66 ± 0.0
XGBoost 0.98 ± 0.0 0.99 ± 0.0 0.42 ± 0.0 0.43 ± 0.0 0.65 ± 0.0
SVM 0.98 ± 0.0 0.99 ± 0.0 0.53 ± 0.0 0.46 ± 0.0 0.72 ± 0.0
AdaBoost 0.89 ± 0.0 0.90 ± 0.0 0.56 ± 0.0 0.20 ± 0.0 0.71 ± 0.0
Proposed model 0.97 ± 0.01 0.98 ± 0.01 0.59 ± 0.1 0.48 ± 0.06 0.76 ± 0.06
FMN
Random Forest 0.84 ± 0.0 0.92 ± 0.0 0.36 ± 0.0 0.29 ± 0.0 0.57 ± 0.0
XGBoost 0.85 ± 0.0 0.93 ± 0.0 0.40 ± 0.0 0.36 ± 0.0 0.61 ± 0.0
SVM 0.69 ± 0.0 0.69 ± 0.0 0.69 ± 0.0 0.28 ± 0.0 0.69 ± 0.0
AdaBoost 0.66 ± 0.0 0.67 ± 0.0 0.64 ± 0.0 0.22 ± 0.0 0.65 ± 0.0
Proposed model 0.88 ± 0.01 0.94 ± 0.01 0.57 ± 0.03 0.51 ± 0.02 0.73 ± 0.02

(Optimal hyperparameters: FAD: Random Forest: num features = 50, num trees = 500, imbalance data handling algorithm (IDHA) = BorderlineSMOTE; XGBoost: learning
rate = 0.1, n_estimators =100, max_depth = 10, early_stopping_rounds = 10, IDHA = Adasyn; SVM: c = 0.01, g = 0.01, IDHA = BorderlineSMOTE; AdaBoost: num features = 500,
IDHA = BorderlineSMOTE; FMN: Random Forest: num features = 100, num trees = 200, IDHA = BorderlineSMOTE or SMOTE; XGBoost: learning rate = 0.1, n_estimators
=1000, max_depth = 5, early_stopping_rounds = 10, IDHA = Adasyn; SVM: c = 0.001, g = 0.01, IDHA = Adasyn; AdaBoost: num features = 100, IDHA = Adasyn).
The highest scores of each metrics are highlighted in bold.

all surveyed algorithms in predicting FAD- and FMN-binding
sites. Furthermore, we obtained sensitivity increases of 6–15%
on FAD and 17–21% if maintaining similar levels of accuracy
and specificity. However, we found that the results of traditional
machine learning algorithms are more stable with standard
deviation being equal to 0. We guess that the reason for this is the
greater number of parameters in the VAEs and CNNs compared
to those from the traditional machine learning algorithms.
Comparison with existing FAD- and FMN-biding
site predictors
Supplementary Table S7, available online at http://bib.oxfordjou
rnals.org/, in the Supplementary data lists previous studies that
predicted FAD- and FMN-binding sites in electron transport
proteins and in general proteins. To make a fair comparison
with respect to prediction performance, we need to use either
the same datasets or the same methods. However, the same
table shows that some of the datasets and working prediction
servers were unavailable when we performed the comparison.
Therefore, we inputted our independent datasets into FADPred
by Mishra et al. [45] and LPIcom by Singh et al. [46] and
reported the performance comparison in Supplementary Table
S8 available online at http://bib.oxfordjournals.org/. We found
that our approach outperformed the previous methods in almost
all metrics.
Reproducing the results
To help reproduce the results, we have freely provided the
VAE_CNN_ETCBinder source codes and data at https://github.
com/khucnam/VAE_CNN_ETCBinder. Using our source codes,
biologists and other researchers can rebuild the model on their
own local machine for prediction. In the near future, we hope
to provide users lacking programming skills with a web-based
prediction server.
Conclusion
In this study, we integrated a novel data imbalance handling
technique based on a VAE with a CNN to improve the identi-
fication of binding sites in electron transport proteins. Using

variational autoencoders and deep neural networks, we were
able to generate new samples that hold important minority class
characteristics due to the great learning ability of the VAE. This
helped us create a high-quality, balanced dataset for training the
model. Apart from variational autoencoders, the convolutional
neural network employed for the classification tasks performed
well on our generated data. We validated our model in a rigorous
manner and found consistent and significant improvements
over all competing approaches for handling data imbalances.
We also greatly improved the ability to identify binding sites (6–
21%) while obtaining a similar or higher level of accuracy and
specificity. Finally, our models show better results compared to
existing predictors for the same binding types, which further
demonstrates the effectiveness of our method. As our input
of the VAEs is position-scoring matrices, this method can be
applied to other protein prediction problems with imbalanced
data. In order to obtain good performance with the neural net-
works, the basic principal for tuning network hyperparame-
ters should be followed (i.e. random or grid search on hyper-
parameters, using learning rate finder or learning rate schedule,
using Keras tuner and applying early-stopping technique) with
the aim of reducing reconstruction loss and the latent loss.
Key Points
• Less than 30% of deep learning–based models for
ligand-binding site prediction address the data imbal-
ance nature of the problem.
• Existing algorithms for handling imbalanced data only
perform linear combinations of minority class sam-
ples.
• This study proposes a new approach for producing
minority class samples by using a variational autoen-
coder and assuming that the added nonlinearity can
be fully exploited by deep neural networks for predic-
tion tasks.
• The effectiveness has been proved by comparison
with conventional data imbalance techniques, tradi-
tional classification algorithms and existing predic-
tors on the same binding types.
Supplementary data
Supplementary data are available online at Briefings in Bioin-
formatics.
Data availability
The surveyed datasets can be downloaded at https://github.
com/khucnam/VAE_CNN_ETCBinder.
Acknowledgment
We would like to thank Ha Phong Nguyen, a Ph.D. candidate
in Oulu University for this helpful comments about our
methodology.
Funding
Ministry of Science and Technology, Taiwan, R.O.C. (MOST
109-2221-E-155-045, MOST 109-2811-E-155-505).
Addressing data imbalance using deep learning 9
References
1. Yang J, Roy A, Zhang Y. BioLiP: a semi-manually curated
database for biologically relevant ligand–protein interac-
tions. Nucleic Acids Res 2012;41(D1):D1096–103.
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
2. Lin W-J, Chen JJ. Class-imbalanced classifiers for high-
dimensional data. Brief Bioinform 2013;14(1):13–26.
3. Chawla NV, Bowyer KW, Hall LO, et al. SMOTE: syn-
thetic minority over-sampling technique. J Artif Intell Res
2002;16:321–57.
4. He H, Bai Y, Garcia EA, et al. ADASYN: adaptive synthetic
sampling approach for imbalanced learning. In: 2008 IEEE
International Joint Conference on Neural Networks (IEEE World
Congress on Computational Intelligence). Hong Kong, China:
IEEE, 2008.
5. Han H, Wang W-Y, Mao B-H. Borderline-SMOTE: a new
over-sampling method in imbalanced data sets learning. In:
International Conference on Intelligent Computing. Hefei, China:
Springer, 2005.
6. Wang Y, Simon M, Bonde P, et al. Prognosis of right ven-
tricular failure in patients with left ventricular assist device
based on decision tree with SMOTE. IEEE Trans Inf Technol
Biomed 2012;16(3):383–90.
7. Nakamura M, Kajiwara Y, Otsuka A, et al. Lvq-smote–
learning vector quantization based synthetic minority over–
sampling technique for biomedical data. BioData Mining
2013;6(1):1–10.
8. Zeng, M., Zou B, Wei F, et al. Effective prediction of three
common diseases by combining SMOTE with Tomek links
technique for imbalanced medical data. In: 2016 IEEE Inter-
national Conference of Online Analysis and Computing Science
(ICOACS). Chongqing, China: IEEE, 2016.
9. Mirza S, Mittal S, Zaman M. Decision support predictive
model for prognosis of diabetes using SMOTE and decision
tree. Int J Appl Eng Res 2018;13(11):9277–82.
10. Ma L, Fan S. CURE-SMOTE algorithm and hybrid algorithm
for feature selection and parameter optimization based on
random forests. BMC Bioinformatics 2017;18(1):1–18.
11. Ishwaran H, O’Brien R. Commentary: the problem of class
imbalance in biomedical data. J Thorac Cardiovasc Surg
2020;1:2.
12. Gao, R., Peng J, Nguyen L, et al. Classification of non-
tumorous facial pigmentation disorders using deep learning
and SMOTE. In: 2019 IEEE International Symposium on Circuits
and Systems (ISCAS). Sapporo, Hokkaido, Japan: IEEE, 2019.
13. Xu Z, Shen D, Nie T, et al. A hybrid sampling algorithm
combining M-SMOTE and ENN based on Random Forest for
medical imbalanced data. J Biomed Inform 2020;107:103465.
14. Wang K-J, Makond B, Chen KH, et al. A hybrid classifier
combining SMOTE with PSO to estimate 5-year survivability
of breast cancer patients. Appl Soft Comput 2014;20:15–24.
15. Abraham B, Nair MS. Computer-aided diagnosis of clinically
significant prostate cancer from MRI images using sparse
autoencoder and random forest classifier. Biocybern Biomed
Eng 2018;38(3):733–44.
16. Kurniawati YE, Permanasari AE, Fauziati S. Adaptive
synthetic-nominal (adasyn-n) and adaptive synthetic-
knn (adasyn-knn) for multiclass imbalance learning on
laboratory test data. In: 2018 4th International Conference on
Science and Technology (ICST). Yogyakarta, Indonesia: IEEE,
2018.
17. Xie C, du R, Ho JWK, et al. Effect of machine learning re-
sampling techniques for imbalanced datasets in 18 F-FDG
PET-based radiomics model on prognostication performance
 10 Nguyen et al.
in cohorts of head and neck cancer patients. Eur J Nucl Med
Mol Imaging 2020;47(12):2826–35.
18. Molinari F, Raghavendra U, Gudigar A, et al. An efficient
data mining framework for the characterization of symp-
tomatic and asymptomatic carotid plaque using bidimen-
sional empirical mode decomposition technique. Med Biol
Eng Comput 2018;56(9):1579–93.
19. Gulrajani I, Kumar K, Ahmed F, et al. Pixelvae: a latent
variable model for natural images. arXiv preprint
arXiv:1611.05013. 2016.
20. Fraccaro M, Sønderby SK, Paquet U, et al. Sequential
neural models with stochastic layers. arXiv preprint
arXiv:1605.07571. 2016.
21. Liu, D. and G. Liu. A transformer-based variational autoen-
coder for sentence generation. In: 2019 International Joint
Conference on Neural Networks (IJCNN). Budapest, Hungary:
IEEE, 2019.
22. Blaschke T, Olivecrona M, Engkvist O, et al. Application of
generative autoencoder in de novo molecular design. Mol
Inform 2018;37(1–2):1700123.
23. Mochel F, Haller RG. Energy deficit in Huntington disease:
why it matters. J Clin Invest 2011;121(2):493–9.
24. Ritov VB, Menshikova EV, Azuma K, et al. Deficiency of
electron transport chain in human skeletal muscle mito-
chondria in type 2 diabetes mellitus and obesity. Am J Physiol
Endocrinol Metab 2010;298(1):E49–58.
25. Barile M, Anna Giancaspero T, Brizio C, et al. Biosynthesis of
flavin cofactors in man: implications in health and disease.
Curr Pharm Des 2013;19(14):2649–75.
26. Lienhart W-D, Gudipati V, Macheroux P. The human flavo-
proteome. Arch Biochem Biophys 2013;535(2):150–62.
27. Liu Y, Fiskum G, Schubert D. Generation of reactive oxygen
species by the mitochondrial electron transport chain. J
Neurochem 2002;80(5):780–7.
28. Gonzalez-Cabo P, Ros S, Palau F. Flavin adenine dinucleotide
rescues the phenotype of frataxin deficiency. PLoS One
2010;5(1):e8872.
29. Vicens Q, Mondragón E, Reyes FE, et al. Structure–activity
relationship of flavin analogues that target the flavin
mononucleotide riboswitch. ACS Chem Biol 2018;13(10):
2908–19.
30. Kuppuraj G, Kruise D, Yura K. Conformational behavior of
flavin adenine dinucleotide: conserved stereochemistry in
bound and free states. J Phys Chem B 2014;118(47):13486–97.
View publication stats
31. Apweiler R, Bairoch A, Wu CH, et al. UniProt: the
universal protein knowledgebase. Nucleic Acids Res
2004;32(suppl_1):D115–9.
32. Altschul SF, Madden TL, Schäffer AA, et al. Gapped BLAST
Downloaded from https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbab277/6329407 by Yuan Ze University, khucnam@yahoo.com on 01 August 2021
and PSI-BLAST: a new generation of protein database search
programs. Nucleic Acids Res 1997;25(17):3389–402.
33. Chen K, Mizianty MJ, Kurgan L. Prediction and analysis
of nucleotide-binding residues using sequence and
sequence-derived structural descriptors. Bioinformatics
2012;28(3):331–41.
34. Chen K, Mizianty MJ, Kurgan L. ATPsite: sequence-based predic-
tion of ATP-binding residues. In: Proteome Science. Hong Kong,
China: BioMed Central, 2011.
35. Lovric M. International Encyclopedia of Statistical Science. Berlin,
Heidelberg, Germany: Springer, 2011.
36. Wang S, Yao X. Using class imbalance learning for software
defect prediction. IEEE Trans Reliab 2013;62(2):434–43.
37. Tang Y, Zhang YQ, Chawla NV, et al. SVMs modeling for
highly imbalanced classification. IEEE Trans Syst Man Cybern
B Cybern 2008;39(1):281–8.
38. Gong J, Kim H. RHSBoost: improving classification
performance in imbalance data. Comput Stat Data Anal
2017;111:1–13.
39. Guo H, Liu H, Wu C, et al. Logistic discrimination based on G-
mean and F-measure for imbalanced problem. J Intell Fuzzy
Syst 2016;31(3):1155–66.
40. Aurelio YS, de Almeida GM, de Castro CL, et al. Learning from
imbalanced data sets with weighted cross-entropy function.
Neural Process Lett 2019;50(2):1937–49.
41. Liu X-Y, Wu J, Zhou Z-H. Exploratory undersampling for
class-imbalance learning. IEEE Trans Syst Man Cybern B
(Cybern) 2008;39(2):539–50.
42. Oh S-H. Error back-propagation algorithm for classification
of imbalanced data. Neurocomputing 2011;74(6):1058–61.
43. Wang S, Minku LL, Yao X. Dealing with multiple classes in online
class imbalance learning. In: IJCAI, Newyork, NY, USA. 2016.
44. Kingma DP, Ba J. Adam: a method for stochastic optimiza-
tion. arXiv preprint arXiv:1412.6980. 2014.
45. Mishra NK, Raghava GP. Prediction of FAD interacting
residues in a protein from its primary sequence using evo-
lutionary information. BMC Bioinformatics 2010;11(1):1–6.
46. Singh H, Srivastava HK, Raghava GP. A web server for anal-
ysis, comparison and prediction of protein ligand binding
sites. Biol Direct 2016;11(1):1–14.