IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO.
9, SEPTEMBER 2023 4409
Multi-Label Local to Global Learning: A Novel
Learning Paradigm for Chest X-Ray
Abnormality Classification
Zean Liu, Yuanzhi Cheng , Member, IEEE, and Shinichi Tamura , Fellow, IEEE
Abstract—Deep neural network (DNN) approaches have
shown remarkable progress in automatic Chest X-rays
classification. However, existing methods use a training
scheme that simultaneously trains all abnormalities without
considering their learning priority. Inspired by the clinical
practice of radiologists progressively recognizing more ab-
normalities and the observation that existing curriculum
learning (CL) methods based on image difficulty may not
be suitable for disease diagnosis, we propose a novel CL
paradigm, named multi-label local to global (ML-LGL). This
approach iteratively trains DNN models on gradually in-
creasing abnormalities within the dataset, i,e, from fewer
abnormalities (local) to more ones (global). At each itera-
tion, we first build the local category by adding high-priority
abnormalities for training, and the abnormality’s priority
is determined by our three proposed clinical knowledge-
leveraged selection functions. Then, images containing ab-
normalities in the local category are gathered to form a new
training set. The model is lastly trained on this set using
a dynamic loss. Additionally, we demonstrate the superi-
ority of ML-LGL from the perspective of the model’s ini-
tial stability during training. Experimental results on three
open-source datasets, PLCO, ChestX-ray14 and CheXpert
show that our proposed learning paradigm outperforms
baselines and achieves comparable results to state-of-the-
art methods. The improved performance promises potential
applications in multi-label Chest X-ray classification.
Index Terms—Chest radiography abnormality classi-
fication, multi-label, clinical knowledge, curriculum
learning, local to global learning.
Manuscript received 3 December 2022; revised 26 March 2023 and
17 May 2023; accepted 25 May 2023. Date of publication 30 May 2023;
date of current version 6 September 2023. The work was supported
in part by the Grants-in-Aid for Scientific Research of Exploratory Re-
search under Grants JP21656100 and JP17K20029. (Corresponding
author: Yuanzhi Cheng)
Zean Liu is with the School of Computer Science and Technology,
Harbin Institute of Technology, Harbin 150001, China (e-mail: zaliuhit@
gmail.com).
Yuanzhi Cheng is with the School of Information Science and Technol-
ogy, Qingdao University of Science and Technology, Qingdao 266061,
China (e-mail: yzcheng2007@163.com).
Shinichi Tamura is with the Department of Radiology, Graduate
School of Medicine, Osaka University, Osaka 565-0871, Japan, also
with the NBL Technovator Company, Ltd., Sennan 590-0522, Japan, and
also with the Rm I-271, ISIR, Osaka University, Osaka 567-0047, Japan
(e-mail: tamuras@nblmt.jp).
Digital Object Identifier 10.1109/JBHI.2023.3281466
2168-2194 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
I. INTRODUCTION
HEST diseases are common but serious health problems
C afflicting many people across the world. For instance,
the COVID-19 pandemic has caused more than 194 million
infections and 4.1 million deaths globally by July 2021 [1].
Due to low cost and easy access, chest X-ray (CXR) is the
most common and important radiological examination used for
early screening and diagnosis. However, these radiographs are
typically interpreted by radiologists, which is labor-intensive
and poses challenges in consistently maintaining high interpre-
tation accuracy. Therefore, developing automatic and accurate
CXR classification algorithms is highly demanded in clinical
applications, and it has become an extensively studied research
area in the medical image community.
With the growing availability of large-scale datasets, deep
neural network (DNN) based methods have delivered best results
among existing methods [3]. These methods typically address
a multi-label classification scenario, attempting to make a set
of binary predictions for each disease. However, the disease
are learned jointly at once during training, ignoring their learn-
ing priority. In clinical scenarios, radiologists’ training process
exhibits a character: they are trained with gradually more ab-
normalities, mirroring the natural human learning process. For
example, as described in [2] and shown in Fig. 1(a), student
trainees prefer to first learn common abnormalities (effusion and
infiltration), and later advance to more rare ones (cardiomegaly
and pneumonia). Such clinical practice motivates us to train
a DNN model with gradually increasing abnormalities, i.e.,
implement a curriculum in the abnormality space.
Aside from clinical inspiration, another persuasive rationale
exists for adopting a curriculum in the abnormality space. While
the curriculum is extensively employed in medical image diag-
nostics, a commonality is that they schedule the presentation
order of training samples for DNN model. For example, [4] fed
images into the model in order of difficulty based on severity
levels mined from radiology reports. However, we argue that
implementing such a curriculum in the sample space may not
be appropriate for disease diagnosis, as the errors made by
a model often arise from confusion between diseases rather
than the absolute difficulty of an image independent of its
class. As exemplified by the confusion matrix in Fig. 1(b),
the model’s mistakes are not uniformly distributed across all
pairs of diseases, and certain diseases, such as fluid and card,
are mainly confused with only a few other diseases. Thus, it
might be more effective to consider the difficulty of disease
instead of that of the images to perform curriculum learning
(CL).
 4410 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO. 9, SEPTEMBER 2023
Fig. 1. (a) An illustration of radiologists’ learning routine, where abnor-
malities are gradually learned from common diseases to rare ones [2];
(b) Confusion matrix of DenseNet121 trained the PLCO dataset. The
position (i, j) indicates the ratio of times the model wrongly classifies an
image as class j instead of the correct class i. The diagonal is removed
to make small differences among the other elements more visible.
Recently, [5] advocated a novel CL, called local to global
learning (LGL), which trains a model from fewer categories
(local) to more categories (global) gradually within the entire
dataset. The learning paradigm is inspired by that humans mem-
orize gradually from fewer categories to more categories. By
simulating such a process, it has been empirically verified that
LGL can avoid local minima and achieve better classification
performance in many tasks [5]. Obviously, LGL is the expected
kind of learning paradigm that can be exploited to achieve our
motivation.
However, LGL is developed for multi-class classification
where a single one-hot prediction is made per image. As more
than one abnormality can be observed on a CXR, we must
operate in a multi-label setting. To this end, we extend LGL
to the multi-label case and propose multi-label local to global
learning (ML-LGL) to iteratively train the DNN model on
gradually increasing abnormalities. We introduce three types of
abnormality selection functions, correlation-based, similarity-
based and frequency-based, to determine abnormality’s learning
priority and create learning orders that resemble radiologists’
progression. Summarizing, our proposed approach results in the
following contributions:
r We propose ML-LGL, an extension of LGL for multi-label
CXR classification. To the best of our knowledge, our
work is the first attempt to investigate the application of
category-based CL for medical image diagnosis, opening
up a myriad of possibilities to explore category-based
CL for improving disease diagnosis. We consider this
pioneering a significant advantage of our approach.
r We introduce three abnormality selection functions to
construct radiologist-like learning order.
r We explain why our ML-LGL works well from the per-
spective of the model’s initial stability during training.
r Experiments are performed on three public datasets,
PLCO, CXR14 and CheXpert. The applicability is demon-
strated and comparable performance is achieved compared
with state-of-the-art approaches.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
II. RELATED WORK
A. Multi-Label CXR Classification
We review the literature of DNN-based methods for CXR
classification, dividing it into the following four groups.
1) General Network-Based Approach: Methods in this part
use vanilla network, simple variants and generic techniques of
deep learning for direct recognition. The first truly large-scale
CXR classification is started by [3], who published a dataset
and introduced a unified framework. Some useful techniques
were subsequently proposed to boost performance, such as dual
architecture [6]. However, these networks still face challenges
related to abnormality complexity, including disease-irrelevant
regions, variable scales, positions, and similarities with sur-
rounding tissue or other abnormalities.
2) Attention-Guided Approach: To alleviate these issues,
attention-guided methods are proposed to extract more dis-
criminative representation. Most methods use attention maps
in different ways to capture disease-critical regions, such as
grad-cam [7], threshold on feature maps [8] and sequential
convolutions [9]. Particularly, channel attention can be used
along with space attention. For example, [10] and [11] utilized
extra channel attention to emphasize the critical disease-specific
features. Moreover, [12] propose channel attention, element
attention and scale attention to simultaneously address these
challenges.
3) Domain Knowledge Integrated Approach: Integrating do-
main knowledge can also acquire more powerful representation.
Methods can be grouped into four families based on the type
of integrated knowledge. 1) Report-based methods incorporate
information from radiological reports, such as [13] and [14],
they both embedded the text context. 2) Additional image-
based methods utilize multiple related images as input. For
example, [15] and [16] used GCN to model image-level and
semantic-level similarities respectively to optimize the visual
feature embedding. 3) Medical knowledge-based methods uti-
lize certain medical knowledge of diseases. For instance, [17]
and [18] proposed a conditional training regime to embed ab-
normality taxonomy. 4) Extra data-based methods use extra data
for joint training, such as [19] and [20], they use a combination
of chest-ray14 and PlCO to perform joint training.
4) Multi-Label Reasoning Approach: These methods are
proposed to capture disease dependency. The dependency can
be constructed in the label space using various ways, such
as LSTM-decoder [21], cascade of binary classifiers [22] and
GCN [23]. The dependency can be also modeled in the fea-
ture space. For example, [11] proposed a spatial and channel
encoding module to strengthen semantic dependencies of multi-
disease features. Recently, [24] introduced a novel semantic-
interactive graph convolutional network (SI-GCN) that inte-
grates concept correlations within each scene and semantic
similarities across different scenes for multilabel image recogni-
tion. Note that our proposed selection function likewise utilizes
the label correlation in each scene and incorporates it into the
curriculum for improved performance. However, we emphasize
that our ML-LGL, as a CL method, is not specifically designed
for addressing the multi-label issue.
B. Related Learning Paradigms for Diagnosis
Our work most directly relates to the fields of curriculum
learning (CL), self-paced learning (SPL), progressive learning
 LIU et al.: MULTI-LABEL LOCAL TO GLOBAL LEARNING
(PL) and brain-like computing. We closely review them in the
medical diagnosis area and explain how they differ from ours.
1) Cl: Using curriculum in the context of machine learning
is first proposed by [25], before [26] coin the term curriculum
learning and trains deep neural network models gradually from
easier to harder data within a dataset. CL has been extensively
applied across various domains, demonstrating improved per-
formance [27]. In medical image diagnosis, CL methods fall
into two categories based on how the curriculum is gener-
ated. 1) Predefined curriculum-based methods manually design
the curriculum using human prior knowledge. For example,
in histopathology image classification, [28] and [29] leverage
annotator agreement and disease severity as a proxy for image
difficulty, respectively. 2) By contrast, transfer teacher-based
methods invite a teacher model and measure training samples’
difficulty based on its output. For example, [30] develop a teacher
model with an evidence identification algorithm, so as to explore
prior knowledge of training bias about diagnosis difficulty and
local features for curriculum generation.
Our ML-LGL conducts training by progressive aggregation of
disease categories within the entire dataset, thereby classifying
it as a form of CL. A key difference between our approach
and other CL methods is curriculum’s implementation space.
While others derive the sample’s curriculum in sample space,
our method adopts a distinct type of curriculum, specified by
establishing the disease’s curriculum in category space.
2) Spl: SPL is the extension of CL that measures the diffi-
culty of training examples according to their losses automati-
cally and embeds the curriculum design into the loss function.
For example, [31] exploit SPL to handle class imbalance of
skin disease, by combining the sample number of each class
and the difficulty of the sample. In multi-modal Alzheimer’s
disease classification [32], SPL is applied to dynamically es-
timate the contribution of each sample to the fusion model
to avoid the influence of noise samples and outliers, and the
sample significance is also helpful to capture the relevance
across different modalities in the multi-modal fusion process.
Apparently, our proposed ML-LGL is out of the scope of SPL,
and more importantly, the curriculum of these methods also takes
place in the sample space.
3) Progressive Learning: “Progressive Learning” is an over-
loaded term used with different meanings. In some literature, it
means a kind of CL method where the curriculum is not related
to the difficulty of each single sample but is configured instead
as a progressive mutation of model capacity or architecture. For
example, [33] progressively decreases the dropout probability
during training and [34] progressively grows the capacity of
GANs to obtain high-quality results. Another meaning of PL is
related to continual or lifelong learning [35], which learns on
increasing tasks using an infinite stream of data.
As a novel kind of CL method, ML-LGL does not belong
to the scope of PL and differs from the aforementioned PL in
two aspects: 1) ML-LGL performs curriculum in the category
space and the model remains unchanged, whereas PL uses a
changeable architecture during the learning process. 2) ML-LGL
uses a fixed dataset set while PL uses increasing and large-scale
data flow.
4) Brain-Like Computing: Brain-like computing mimics the
information processing mode and structure of the biological
nervous system, proposing new computing theory, computer
architecture and learning algorithms [36]. In medical diagnosis,
the methods can be roughly divided into two families. 1) The
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
4411
network architecture-based methods primarily use bio-inspired
spiking neural networks (SNNs) [37] in various tasks. For
example, [38] proposes an SNN-based framework to mitigate
the class imbalanced problems in medical image classification,
while [39] uses the SNNs for COVID-19 detection. 2) The
learning paradigm-based methods aim to mimic how humans
learn in the real world. Representative works include transfer
learning, few-shot learning [40] and the above-mentioned CL.
Our proposed ML-LGL method is also inspired by how people
learn, but it specifically focuses on the mechanism of how
individuals learn from a few categories to more categories.
III. PRELIMINARIES: LGL
Consider the problem of multi-class classification on cat-
egories Y using a DNN model, where w denotes the model
weights and L denotes the loss function. The definition of LGL
introduced in [5] can be given as follows:
Definition 1: The LGL methodology is to iteratively train the
DNN model by adding a new category’s samples to the training
set at each time.
LGL is implemented by iteratively minimizing the loss func-
tion on gradually increasing training sets. At the k-th iteration,
let Ykl denotes the local category where the categories have
already been trained, and wk∗ denotes convergent weights. At
each iteration, the implementation consists of three steps:
1) Update the local category: Using the category selection
function f to select a new category yks from the remaining
untrained categories, and push it into the local category
to form a new set:

yks = f (Y\Yk−1
l
)
(1)
Yk = Yk−1 ∪ {yks }
l l
2) Build a new training set: Select samples whose labels
are in the new local category to build the new training
set tk :
tk = {(xn , yn ) : yn ∈ Ykl } (2)
3) Training: Train the model on the new training set tk using
the loss function L until convergence:
wk∗ = arg min L(w; tk , wk−1
∗
)
w (3)
IV. METHOD
Building on LGL, we propose ML-LGL for multi-label
CXR classification. Given the dataset Dcxr = {(xn , Yn )}N
n=1
with K abnormalities Y = {y1 , y2 , . . .yK }, xn is an input
image, and Yn is its corresponding label, which is a subset
of Y. The flowchart of ML-LGL is shown in Fig. 2, and
Algorithm 1 outlines the process. ML-LGL follows the general
steps of LGL: selecting new disease patterns to update the
local category, building a new training set, and training on
the new set using dynamic loss. We will now delve into the
details.
A. Selection Function for Local Category Updating
The core of updating local category at each iteration is to
use the selection function f to choose diseases of high training
priority. To enable the model to learn like a radiologist, f must
 4412 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO. 9, SEPTEMBER 2023

Fig. 2. ML-LGL for multi-label CXR classification, which iteratively trains the DNN model on gradually increasing abnormalities. It shows the three
steps involved in each iteration: select new disease to update the local category, build a new training set, and train on the set using dynamic loss.

Fig. 3. Illustration of three clinical knoweledge-leveraged selection functions: (a) co-occurrence frequency matrix of 12 disease patterns from
the PLCO dataset; (b) multi-label conditional entropy computing; (c) frequency of disease pattern from PLCO. The 12 disease are nodule, mass,
pleural-based mass, granuloma, fluid in pleural space, hilar, infiltration, scarring, pleural fibrosis, bone lesion, cardiac abnormality, and COPD.

characterize radiologists’ learning patterns. Accordingly, we
propose three selection functions that leverage clinical knowl-
edge to establish radiologist-like learning order.
1) Correlation Function: Radiologists typically determine
the learning order based on prior knowledge of comorbidity
(Fig. 3(a)). They tend to prioritize learning diseases that are
strongly related to other ones, as this enables them to leverage
as much relevant experience gained from previously studied
diseases as possible when diagnosing subsequent ailments.
Therefore, we aim to assign higher training priority to strongly
correlated diseases.
To achieve this, we use the co-occurrence frequency of two
diseases to measure their correlation and use it to compute the
total correlation for each disease. The disease with higher total
correlation is selected first to be pushed into the local category
for training. The correlation function is given by:
K
 2 × #(yi , yj )
fcor = arg max (4)
l
yi ∈Y\Yk−1 #(yi ) + #(yj )
j=1,j=i
where #(yi ) represents the number of samples where the disease
yi exists, and #(yi , yj ) represents the number of samples where
both diseases yi and yj co-exist.
2) Similarity Function: Radiologists tend to prioritize learn-
ing about similar diseases based on their historical learning
experiences, which motivates us to assign a higher training
priority to diseases more similar to those already learned
To achieve this, we propose multi-label conditional entropy
(MLCE) as a similarity metric. As shown in Fig. 3(b), given

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
 LIU et al.: MULTI-LABEL LOCAL TO GLOBAL LEARNING
the model weights w and image dataset X, the MLCE can be
computed as follows:

Hw (O | X) = p(x)Hw (O | X = x)
x∈X
N K
1  k
=− [on log2 okn +(1−okn )log2 (1−okn )]
N n=1
k=1
(5)
where O is the output of classification layer and okn is the sigmoid
output of the k-th node for image xn .
We quantify the similarity between each untrained disease
with the local category. The one with the highest similarity value
is first selected. Let Xyi represent the image labeled solely with
yi , the similarity function is given as:
fsim = arg max Hwk−1 (O | Xyi )
l
yi ∈Y\Yk−1
Note that the initial disease is chosen based on weights trans-
ferred from ImageNet.
3) Frequency Function: As mentioned in the introduction,
radiologists typically start by learning commonly occurring
disease patterns and gradually move on to rare ones over time.
To emulate this approach, we introduce the frequency function:
ff re = arg max #(yi )
l
yi ∈Y\Yk−1
This function prioritizes disease patterns with high frequency.
B. Build New Training Set
The updated local category is then used to construct the
training set for current iteration. In the multi-class case, the
dataset is obtained by selecting samples whose labels are in the
local category (see (2)). However, in our multi-label case, this
operation is not applicable because some samples have multiple
labels, part of which does not belong to the local category. As
an alternative, we select images whose labels intersect with the
local category to create a new training set:
tk = {(xn , Yn ) : Yn ∩ Ykt = ∅}
C. Training on Dynamic Loss
We use DenseNet121 as the backbone to train the DNN
model on the new training set. The original fully connected and
classification layer are removed and a fully-connected layer of
K neurons and a classification layer with sigmoid function are
added. Each output represents the likelihood of belonging to
corresponding disease pattern and the cross-entropy loss is used
for each disease pattern. To train the model only on the local
category, we control the loss function using the characteristic
function, resulting in a dynamic loss function:
 Third, our motivation for developing ML-LGL is more pro-
L(w; tk , wk−1 ) = CE(zy , z(y)) ∗ 1y∈Ykt (9)
y∈Y
where CE(·, ·) denote the cross-entopy loss, y denotes the
disease pattern, and z(y) ∈ {0, 1} denotes its ground truth with
z(y) corresponding to model’s sigmoid output.
To speed up training, multiple disease patterns can be added
simultaneously in each iteration, referred to as a “bucket”. The
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
4413
Algorithm 1: ML-LGL for CXR Classfication.
(6)
number of buckets is denoted as nb , and the number of disease
patterns added in the k-th iteration can be calculated using:

K/nb k < nb
add(k) = (10)
K − (nb − 1) × K/nb k = nb
This is more efficient than adding disease patterns one by one,
(7) which is time-consuming due to the linear relationship between
the training time and the number of disease patterns (K) as
proven in [5], as well as the time required for abnormality
selection, especially for the dynamic similarity function that
involves feedforward estimating of images.
D. Difference With LGL
Although our proposed ML-LGL shares similarities with
original LGL, there are several key differences that highlight
the novelty of our method. First, LGL was originally developed
for single-label classification and tested on single-label natural
image datasets. However, the multi-label setting of CXR makes
it unavailable for direct utilization. To achieve our objective of
developing DNN model for CXR abnormality classification, we
(8) must consider the multi-label setting in our implementations,
such as the new training set acquiring and loss function. Conse-
quently, we utilize LGL as a solid foundation for our ML-LGL,
adapting it to accommodate the specific requirements of the
multi-label context.
Second, our ML-LGL provides more elaborated selection
functions that embody the inherent characteristics of CXR
datasets. For instance, the correlation function considers label
relations, yielding improved performance in experiments. In
contrast, the original LGL focused on proposing the overall
learning paradigm and only tested with two simple and universal
category selection functions (random and similarity).
found. As presented in the introduction, excluding being in-
spired by radiologists’ training processes, our ML-LGL is also
motivated by the observation that model errors often stem from
confused disease classes rather than independent samples, which
motivates us to implement category-based CL in medical image
diagnosis for the first time. In contrast, LGL is only inspired
by the human learning mechanism of gradually learning more
categories.
 4414 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO. 9, SEPTEMBER 2023
Fig. 4. (a) Initial stability across varying disease numbers; (b) AUC
performance at the convergent state for different disease numbers.
Overall, our ML-LGL approach is a novel and effective solu-
tion for multi-label CXR abnormality classification, with unique
features that distinguish it from LGL.
V. EXPLANATION OF MODEL STABILITY PERSPECTIVE
To understand why ML-LGL works well, we give an expla-
nation from the perspective of model stability during training.
First, we define model stability as follows:
Definition 2: Given the image dataset X and model weights
w, we use the multi-label conditional entropy (MLCE) defined
in (5) to quantify the model stability. A lower MLCE signifies a
more stable model.
We then devise a series of comparative experiments using
the PLCO dataset. A non-ML-LGL model is established, which
shares the same DenseNet121 backbone as the ML-LGL model
but employs random initialization for training. The ML-LGL
model is trained using nb = 12 and the correlation function,
while the non-ML-LGL model is trained on a series of updated
datasets established by the ML-LGL model (see (8)). We report
the stability at the initial moment and the AUC performance
at the convergent moment for each training session of the two
models. Both metrics are evaluated on the corresponding subset
of the validation set, i.e., the image-label pairs whose labels
intersect with the currently trained diseases. Consequently, we
obtain the metric values on various disease numbers.
As shown in Fig. 4, the stability of ML-LGL model tends
to be unchanged when the disease number exceeds 8, whereas
the non-ML-LGL model demonstrates sustained growth with a
slope of 1. More importantly, the stability gap and the AUC gap
between the two models synchronously expand as the disease
number increases, indicating a positive correlation between the
reduced initial stability achieved by ML-LGL and the AUC
performance. Therefore, we infer that ML-LGL benefits the
training process by lowering the initial stability, ultimately
attaining improved performance. Note that our explanation is
provided from the perspective of initial stability, grounded in
experimental validation. More theoretical explanations could be
given from continuation method [41] or data distribution [42].
VI. EXPERIMENT AND RESULTS
A. Dataset
We conduct experiments on three public available CXR
datasets: PLCO [43], ChestX-ray14 [3] and CheXpert [44].
Table I gives an overview.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
TABLE I
OVERVIEW OF THE THREE DATASETS
1) PLCO: PLCO is a CXR arm of the National Cancer Insti-
tute’s screening trial for prostate, lung, colorectal, and ovarian
cancer [43]. There are 198,000 images in total but only 84182
images from 25 k participants are publicly distributed, with 13
disease patterns labeled by radiologists’ visual observation. We
exclude major atelectasis due to its low prevalence and use the
remaining 12 ones for experiments. Without provided official
split, we split the data into three patient-level subgroups: 70%
for training, 10% for validation, and 20% for testing, maintaining
disease pattern prevalence in each subgroup.
2) ChestX-Ray14: ChestX-ray14 (CXR14), published by
the US National Institute of Health, is the most widely used
benchmark in the field of CXR analysis. The 14 labels are
minded from radiological reports by using natural language pro-
cessing (NLP). To establish a consistent benchmark, we follow
the official patient-wise split standard [3]. During training, we
use 10% of images from training set for validation.
3) CheXpert: CheXpet is a large-scale dataset released by
Stanford Hospital and here we use the 1.0 version dataset with
downsampled resolution. The labels of training set are automati-
cally extracted from radiology reports using a rule-based labeler,
while the validation and test sets are manually annotated by the
consensus of board-certified radiologists. 12 common disease
patterns are labeled as positive, negative and uncertain. The test
set is not publicly available and we evaluate our method on the
validation set. We adopt two commonly utilized uncertain label
policies [44] to use uncertain labels: 1) U-Zeros that replaces all
the uncertain labels with “zero” and 2) U-Ones that replaces all
the uncertain labels with “one”.
4) Remarks: Label Noise: The CXR14 and CheXpert
datasets use NLP to extract labels from reports, which can result
in label noise for two main reasons. First, NLP has high levels of
error and uncertainty [44]. Second, text reports cannot replace
visual examination of the image due to insufficient or overly
descriptive information (e.g., lab tests or prior radiological stud-
ies). Label noise is severe in CXR14, with positive predictions
mostly 10% to 30% lower than original values [45]. As a result,
PLCO is the only large-scale CXR dataset with labels produced
by radiologists’ visual observation of CXR images.
B. Experimental Setup
1) Evaluation: The per-class area under the ROC curve
(AUC) is adopted to measure the performance of each disease
pattern, and the average AUC is employed to evaluate over-
all performance. In our experiments, differences in the two
compared methods are assessed using the one-sided DeLong
statistical test [46] and the p-value < 0.05 is considered as the
 LIU et al.: MULTI-LABEL LOCAL TO GLOBAL LEARNING
TABLE II
THE AUC SCORES OF VARIOUS MODELS ON PLCO
Fig. 5. Comparison of mean AUC evaluated on PLCO with different nb .
significant level for all statistical tests. The statistical test is
performed using Matlab implementation1.1
2) Training Details: During training, the raw images of
PLCO and CXR14 are resized to 512 × 512 via down-sample
and random crop, while images of CheXpert are directly cropped
to 320 × 320. The processed images are normalized with mean
and standard deviation of ImageNet images. We only use hori-
zontal flipping to augment training data.
The framework is trained by an Adam optimizer with default
β parameters (β1 = 0.9, β2 = 0.99), a weight decay of 0.01, and
an adaptive learning rate. The strategy proposed in [47] is used
to find the proper initial learning rate at each iteration, and the
learning rate is reduced by 5 times when the mean AUC on the
validation set plateaus for more than 5 epochs. The mean AUC on
the validation set is also used to terminate the training procedure
when it plateaus for more than 10 epochs, and the model with
the highest mean AUC on validation set will be the final model.
For validation and inference, we crop 512 × 512 (PLCO and
CXR14) and 320 × 320 (CheXpert) sub-images (a central one
and four corner ones) as the network input. Our experiments
are implemented with PyTorch on two 2080Ti GPUs of 12 GB
memory, and the batch size is set to 32 (CheXpert) and 16
(CXR14 and PLCO).
C. Parameter Analysis for Bucket Number
We investigated the impact of the bucket number on the
performance of ML-LGL by testing different nb values, namely
nb = 2, 3, 4, 5, using three selection functions. The experiments
were conducted on PLCO, which provides more reliable results
1 https://github.com/PamixSun/DeLongUI
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
4415
due to its cleanliness. The results presented in Fig. 5 demonstrate
that increasing nb leads to a continuous improvement in the mean
AUC of all three functions. However, the growth rate is sublinear.
As mentioned in Section IV-C, larger nb requires more training
time, indicating a need to balance performance and training time.
In subsequent experiments, we set nb = 4 for PLCO and CXR14
and nb = 3 for CheXpert.
D. Comparison With the Baseline
The mean AUC of the three functions outperforms the base-
line on all three datasets, as shown in Tables II–IV. Per-class
AUC indicates significant improvement on over half of disease
patterns on their respective datasets, except for the frequency
function on PLCO. These findings demonstrate the effectiveness
of ML-LGL in enhancing CXR classification performance. Ab-
normality heatmaps, generated by grad-cam, visually illustrate
this effectiveness. In the single-label sample (Fig. 7(a)), all three
functions increase the likelihood of mass and improve the match
between the identified abnormality area and the bounding box,
particularly for the correlation function. In the two-label sample
(Fig. 7(b)), the similarity and correlation functions correctly
diagnose cardiomegaly and indicate well-matched abnormality
areas, while the baseline predicts no cardiomegaly and mis-
diagnoses effusion with a likelihood of 0.730. However, not
all disease predictions show significant improvement, such as
infiltration, fibrosis and pneumonia on CXR14, as depicted in
Fig. 7(c). For infiltration, the three functions demonstrate a
likelihood as low as the baseline and no high activation values
in the corresponding bounding box, despite higher likelihood
and more precise disease localization for its co-existing diseases
(atelectasis and effusion).
E. Comparison of Three Selection Functions
Results vary across three functions according to Tables II–IV.
On PLCO, correlation and similarity functions have comparable
mean AUC and both significantly outperform the frequency
function on most diseases, with correlation function showing
notable improvement in 7 out of 12 diseases. On CXR14,
mean AUC difference of three functions is not noticeable, with
only 4 out of 14 diseases showing a statistical significance
between frequency function and correlation/similarity function.
On CheXpert, the difference is significant for 13 diseases but not
for 5. We attribute the performance difference for three functions
 4416 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO. 9, SEPTEMBER 2023

TABLE III
THE AUC SCORES OF VARIOUS MODELS ON CXR14

TABLE IV
THE AUC SCORES OF VARIOUS MODELS ON CHEXPERT

not only to inherent variations among datasets but also to label
noise, which we will explore in Section VII-D.
F. Compare With State-of-The-Art Methods
We compare our method against state-of the-art methods on
the three datasets and Fig. 6 shows ROC of our best results.
1) Results on PLCO: We only compared our method with
only HMLC [17] due to limited studies on this dataset. [20]
also experimented on PLCO but used a previously pub-
lished dataset with up to 198000 images, which is now
not available due to changes in the data release policy by
the organizers. Moreover, this method reported results us-
ing training data that included samples from CXR14. It is
also not possible to make a direct comparison with HMLC
regarding the per-class AUC score, as they use different
data splits. Thus, to ensure fairness, we only compared the
overall performance. As shown in Table II, our correla-
tion function outperforms HMLC, and the similarity function
achieves comparative results, demonstrating the effectiveness of
ML-LGL.
2) Results on CXR14: We select 15 typical and well-
performed methods that are evaluated on the official patient-wise
split. To provide more comprehensive comparisons, we attempt
to compare our proposed method with SI-GCN by [24], which

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
 LIU et al.: MULTI-LABEL LOCAL TO GLOBAL LEARNING
Fig. 6. The ROC curve of our best results from the three datasets.
has shown superior performance on multi-label natural images.
However, the results are obtained from different datasets and
thus cannot be used for reliably comparing the performance of
the two methods. Fortunately, we note that CheXGCN by [23]
and SSGE by [16] are two special cases of SI-GCN tailored
for CXR classification. Specifically, CheXGCN and SSGE both
mirror the corresponding components of SI-GCN to build con-
cept correlations of the same scene and the semantic similarities
of different scenes, respectively. Therefore, we use the results
reported in [16], [23] for our comparison.
As shown in Table III, our similarity function achieves the
second-highest mean AUC, narrowly beaten by [19] who used
CheXpert as an external dataset for joint training. Notably, our
similarity function performed comparably with the previous
best results, [48] and SSGE [16], both of which used only
CXR14. [48] proposed squeeze-and-excitation blocks, multi-
map transfer and max-min pooling to learn disease-specific fea-
tures, thus achieving good performance. SSGE [16] explored the
semantic similarities of in-batch images to optimize the feature
embedding, also leading to good results. As previous best work
that exploits label correlations, CheXGCN [23] compares fairly
against our frequency function, but is inferior to our correlation
and similarity function. These observations demonstrate the
superiority of our proposed ML-LGL.
Compared to the multi-modal learning method TieNet [13],
which uses additional reports, and the multi-view method Im-
ageGCN [15], which uses multi-relational images (e.g., from the
same age and person), three functions we proposed can consis-
tently achieve the highest mean AUC with higher per-class AUC
on almost all diseases. Furthermore, it outperforms AG-CL [4],
which also uses CL for training, by a significant margin. These
comparisons again persuasively demonstrate the advantage of
our proposed ML-LGL.
3) Results on CheXpert: For a fair comparison, we selected
state-of-the-art methods that were evaluated on the official val-
idation set. It should be noted that although the training dataset
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
4417
consists of 14 observations, the validation set only contains
13 diseases, with no samples of fracture, and some methods
only evaluated the 5 clinically important and prevalent diseases.
Therefore, we reported the average AUC for both 5 and 13 types
of diseases simultaneously.
We first examine the performance of the correlation function
on the 5 diseases. Table IV show that the mean AUC achieved
by our correlation function is comparable to the previous best
method, CheXpert [44], in the U-zeros setting, despite CheXpert
using an ensemble of 30 models. Moreover, our correlation
function ranks first in the U-ones setting.
Moving on to the performance on the 13 diseases, we find that
our correlation function achieves the best overall performance
and shows the top performance on more than half of the disease
in both two-label settings, outperforming the other two multi-
view learning methods: MVC-Net [49] and ImageGCN [15],
which take multi-view images (i.e., frontal and lateral) and multi-
relational images as input, respectively.
Note: The original results of ImageGCN on both CXR14 and
CheXpert datasets, as well as MVC-Net on CheXpert, were
reported on their respective data splits. We have reported the
results using the released code.2 In this comparison, ImageGCN
employed DenseNet121 as the backbone model.
VII. ABLATION STUDY
A. Is Radiologists-Like Learning Order Beneficial?
To examine how much radiologist-like learning order con-
tribute to ML-LGL, we give a new selection function for compar-
ative study called random function, which selects abnormalities
randomly at each iteration. Table II shows random function also
achieves a higher mean AUC than the baseline, demonstrating
that ML-LGL can naturally benefit CXR classification regard-
less of learning order. This finding supports the motivation of
LGL [5] and our explanation in Section V.
We compared the random function with the three clinical
knowledge-leveraged functions in terms of mean AUC and
per-class AUC. We found that the random function achieved
results similar to the frequency function, but was outperformed
by the correlation and similarity functions. This suggests that the
correlation and similarity order can benefit the training process,
while the frequency order cannot. The similarity order makes the
model start training from the most stable state at each iteration as
demonstrated in Section V, which may possibly explain the su-
perior performance. The improvement brought by the correlation
function may be attributed to the exploitation of label relations,
which is consistent with previous methods [11], [23] that have
demonstrated the boosting of performance through label relation
modeling.
B. Deep Analysis of Radiologists-Like Learning Order
To aetiologically understand the radiologists-like learning
order, We tested the reverse versions of the correlation and
similarity functions. The anti-correlation function learns weaker
correlated disease patterns first, while the anti-similarity func-
tion learns dissimilar disease patterns first. The reverse version
of the frequency function is not tested as it does not improve
performance.
2 https://github.com/fzfs/Multi-view-Chest-X-ray-Classification; https://
github.com/mocherson/ImageGCN
 4418 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO. 9, SEPTEMBER 2023

Fig. 7. Classification results visualization from CXR14. Image with one-label (a), two-label (b) and three-label (c) are shown. From top to bottom:
heatmaps generated by baseline and three selection functions. The highest five sigmoid values are shown with ground truth highlighted in red.

TABLE V representation power (from VGG16 to AG-CNN). Both selec-

RESULTS OF DIFFERENT DNN MODELS ON THE PLCO DATASET tion functions yielded mean AUC improvement for all models
compared to their respective baselines, indicating the effective-
ness of our approach for DNN models with different levels of
complexity.

Comparing the results in Table II, the reverse versions
had significantly different outcomes compared to their orig-
inal counterparts. Surprisingly, the anti-similarity function
showes a higher average AUC and improved 5 disease pat-
terns compared to its original version. This may be because
the similarity function prioritizes disease with greater sim-
ilarity, which are more informative and further away from
the currently trained images. Training on these images first
can rapidly reduce the error and lead to better outcomes.
Conversely, the anti-correlation function decreased the mean
AUC drastically and approached the baseline, indicating that
it is crucial to learn strongly correlated disease patterns first
and incorrect learning order may adversely affect the training
process.
C. Generalizability on Different DNN Models
To validate ML-LGL’s generalizability across models, an-
other three DNN models are involved for evaluation: VGG16,
ResNet50 and AG-CNN [8]. The experiments were conducted
on PLCO using correlation and similarity functions, and AG-
CNN was implemented using code3 provided by the authors. As
shown in Table V, the performance increased with the model’s
3 https://github.com/Ien001/AG-CNN/blob/master/
D. Limitation
Our approach is susceptible to label noise due to the learn-
ing order built on the label, causing the clinical knowledge-
leveraged function to degrade into a random function and hinder
learning ability. Our experimental results in Tables III and IV,
which show no significant difference in the mean AUC of the
three functions, seem to support this hypothesis.
To demonstrate our hypothesis, we perform experiments to
simulate different levels of label noise severity to monitor per-
formance change. Since PLCO has greater label reliability, we
regard it as a basic dataset to simulate a label noise scenario
that CXR14 and CheXpert face. Label noise specifically refers
to three cases: missed finding, over-labeling and mislabeling.
Missed finding means an abnormality is observed on a CXR
while it is not annotated. Over-labeling means an abnormality
does not exist on a CXR but it is labeled as positive. Mislabeling
means an abnormality is misidentified and labeled as another
name. Accordingly, we corrupt PLCO to produce the scenario
using the following controlled scheme:
r Choose a base probability α ∈ [0, 0.25]
r For each sample with abnormality, we randomly select one
of its labels and delete it with a probability of α.
r For each sample with abnormality, we replace one of its
labels with another label (not originally included in the
sample) with a probability of α.
r For each sample, we add an extra label (not originally
included in the sample) with a probability of α.

Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
 LIU et al.: MULTI-LABEL LOCAL TO GLOBAL LEARNING
Fig. 8. Mean AUC with with 95% confidence interval on different levels
of label noise severity using various selection functions.
r The previous operation is overlayed by the subsequent one
on each sample.
We tested α = 0, 0.05, 0.1, 0.15, 0.2, 0.25 using original and
reverse versions of the correlation and similarity functions, with
the random function also synchronously tested for comparison.
To allow comparisons across α, we ensured that if a label was
operated on at a certain α value, it would also be operated on all
larger α values.
As noise severity increases, overall performance drops for
all three functions as shown in Fig. 8, confirming the negative
impact of label noise. The anti-similarity function consistently
outperforms the similarity function, and the correlation function
also performs continually better than the ant-correlation func-
tion. Notably, as the severity of noise increases, the performance
gap between the original, reverse and random functions shrinks,
with the original and reverse functions approaching the perfor-
mance of the random function when α = 0.25. These results
provide further evidence for our hypothesis.
VIII. DISCUSSION AND CONCLUSION
In this paper, we propose ML-LGL, a novel learning paradigm
tailored for multi-label CXR classification. It trains DNN model
with a curriculum of gradually increasing abnormalities, which
is generated by clinical knowledge-leveraged selection func-
tions. ML-LGL benefits training process by initiating models
from a more stable state. Our evaluations demonstrate com-
parable performance with state-of-the-art methods on PLCO,
CXR14, and CheXpert datasets. The proposed learning orders
of correlation and similarity improved training, and their reverse
orders yielded different results. Our experiments with different
levels of label noise severity indicated that ML-LGL lacks
robustness to label noise.
Our ML-LGL is a category space-based CL method and
is model-agnostic, making it applicable to all existing multi-
modal CXR classification methods, such as ImageGCN [15],
TieNet [13] and MVC-Net [49], irrespective of the fusion tech-
niques used. Concerning its extendability to multimodal learn-
ing beyond medical image diagnosis. First, while multimodal
learning involves various tasks (segmentation, translation and
alignment), ML-LGL is currently only extendable to classifi-
cation models. Further research will explore the possibility of
extending to other tasks. Second, ML-LGL can only be applied
in the category space, and not in the sample space [32] or
modality space [50].
There are several interesting avenues for future work. First,
we could explore other radiologist-like learning orders, such as
hierarchical curriculum based on abnormality taxonomy [17].
Second, instead of adding an equal number of diseases at each
iteration, we could use clustering algorithms to automatically
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
4419
generate more compact buckets to improve performance, as
described in [51]. Third, while this paper only introduces
radiologist-like curricula, it is flexible enough to support
curricula that extend beyond mimicking radiologists. For
example, the confusion matrix depicted in Fig. 1(b) can be
used to develop a hierarchical curriculum where diseases are
trained from coarse to fine levels. Specifically, the model that
is directly trained on the training set is first used to estimate the
matrix. Then, the matrix is input into a hierarchical clustering
algorithm to construct the disease hierarchy, where diseases
that are more frequently confused are gathered at lower levels.
Such a heuristic curriculum does not require the expertise of
radiologists and may yield even greater improvements.
Considering the potential applicability of ML-LGL in clin-
ical settings, it is crucial to pay attention to situations where
the model can be attacked by imperceptible and carefully-
engineered adversarial examples. To prepare for this, several
aspects can be investigated in the future. First, we need to
evaluate the adversarial robustness of the model. Unlike assess-
ments on natural images and other medical diagnosis tasks as
discussed in [52], our evaluation must consider the factor of
label co-occurrence and label noise. We also need to perform
the evaluation on different datasets using various threat models
(white box, gray box and black box) and attacks of different in-
tensities. Second, we need to explore how adversarial examples
attack the model and identify its vulnerability. Basic exploration
techniques could include analyzing intermediate features and
estimating label correlations, for both original/clean images
and adversarial ones, as introduced by [53]. Third, based on
the above analysis, we need to design an easy-to-deploy and
effective defense method to handle adversarial examples. Doing
so will improve the robustness of the model against adversarial
attacks, and therefore increase its utility in clinical settings.
ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers
for their valuable comments and help suggestions that greatly
improved the paper’s quality.
REFERENCES
[1] M. Chavez-MacGregor, X. Lei, H. Zhao, P. Scheet, and S. H. Giordano,
“Evaluation of COVID-19 mortality and adverse outcomes in us patients
with or without cancer,” JAMA Oncol., vol. 8, no. 1, pp. 69–78, 2022.
[2] S. Sait and M. Tombs, “Teaching medical students how to interpret chest
X-rays: The design and development of an e-learning resource,” Adv. Med.
Educ. Pract., vol. 12, 2021, Art. no. 123.
[3] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “ChestX-
ray8: Hospital-scale chest X-ray database and benchmarks on weakly-
supervised classification and localization of common thorax diseases,” in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2097–2106.
[4] Y. Tang, X. Wang, A. P. Harrison, L. Lu, J. Xiao, and R. M. Summers,
“Attention-guided curriculum learning for weakly supervised classifica-
tion and localization of thoracic diseases on chest radiographs,” in Proc.
Int. Workshop Mach. Learn. Med. Imag., Springer, 2018, pp. 249–258.
[5] H. Cheng, D. Lian, B. Deng, S. Gao, T. Tan, and Y. Geng, “Local to global
learning: Gradually adding classes for training deep neural networks,” in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 4748–4756.
[6] B. Chen, J. Li, X. Guo, and G. Lu, “DualCheXNet: Dual asymmetric fea-
ture learning for thoracic disease classification in chest X-rays,” Biomed.
Signal Process. Control, vol. 53, 2019, Art. no. 101554.
[7] H. Wang, H. Jia, L. Lu, and Y. Xia, “Thorax-Net: An attention regular-
ized deep neural network for classification of thoracic diseases on chest
radiography,” IEEE J. Biomed. Health Inform., vol. 24, no. 2, pp. 475–485,
Feb. 2020.
 4420 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 27, NO. 9, SEPTEMBER 2023
[8] Q. Guan, Y. Huang, Z. Zhong, Z. Zheng, L. Zheng, and Y. Yang, “Diagnose
like a radiologist: Attention guided convolutional neural network for thorax
disease classification,” 2018, arXiv:1801.09927.
[9] Q. Guan and Y. Huang, “Multi-label chest X-ray image classification
via category-wise residual attention learning,” Pattern Recognit. Lett.,
vol. 130, pp. 259–266, 2018.
[10] B. Chen, J. Li, G. Lu, and D. Zhang, “Lesion location attention guided
network for multi-label thoracic disease classification in chest X-rays,”
IEEE J. Biomed. Health Inform., vol. 24, no. 7, pp. 2016–2027, Jul. 2020.
[11] Q. Guan, Y. Huang, Y. Luo, P. Liu, M. Xu, and Y. Yang, “Discriminative
feature learning for thorax disease classification in chest X-ray images,”
IEEE Trans. Image Process., vol. 30, pp. 2476–2487, 2021.
[12] H. Wang, S. Wang, Z. Qin, Y. Zhang, R. Li, and Y. Xia, “Triple attention
learning for classification of 14 thoracic diseases using chest radiography,”
Med. Image Anal., vol. 67, 2021, Art. no. 101846.
[13] X. Wang, Y. Peng, L. Lu, Z. Lu, and R. M. Summers, “TieNet: Text-image
embedding network for common thorax disease classification and report-
ing in chest X-rays,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,
2018, pp. 9049–9058.
[14] G. Jacenków, A. Q. O’Neil, and S. A. Tsaftaris, “Indication as prior
knowledge for multimodal disease classification in chest radiographs with
transformers,” in Proc. IEEE 19th Int. Symp. Biomed. Imag., 2022, pp. 1–5.
[15] C. Mao, L. Yao, and Y. Luo, “ImageGCN: Multi-relational image graph
convolutional networks for disease identification with chest X-rays,” IEEE
Trans. Med. Imag., vol. 41, no. 8, pp. 1990–2003, Aug. 2022.
[16] B. Chen, Z. Zhang, Y. Li, G. Lu, and D. Zhang, “Multi-label chest X-ray im-
age classification via semantic similarity graph embedding,” IEEE Trans.
Circuits Syst. Video Technol., vol. 32, no. 4, pp. 2455–2468, Apr. 2022.
[17] H. Chen, S. Miao, D. Xu, G. D. Hager, and A. P. Harrison, “Deep
hiearchical multi-label classification applied to chest X-ray abnormality
taxonomies,” Med. Image Anal., vol. 66, 2020, Art. no. 101811.
[18] H. H. Pham, T. T. Le, D. Q. Tran, D. T. Ngo, and H. Q. Nguyen, “Interpret-
ing chest X-rays via CNNs that exploit hierarchical disease dependencies
and uncertainty labels,” Neurocomputing, vol. 437, pp. 186–194, 2021.
[19] L. Luo et al., “Deep mining external imperfect data for chest X-ray disease
screening,” IEEE Trans. Med. Imag., vol. 39, no. 11, pp. 3583–3594,
Nov. 2020.
[20] S. Gündel et al., “Robust classification from noisy labels: Integrating
additional knowledge for chest radiography abnormality assessment,”
Med. Image Anal., vol. 72, 2021, Art. no. 102087.
[21] L. Yao, E. Poblenz, D. Dagunts, B. Covington, D. Bernard, and K. Lyman,
“Learning to diagnose from scratch by exploiting dependencies among
labels,” 2017, arXiv:1710.10501.
[22] P. Kumar, M. Grewal, and M.M. Srivastava, “Boosted cascaded convnet
for multilabel classification of thoracic diseases in chest radiographs,” in
Proc. Int. Conf. Image Anal. Recognit., Springer, 2018, pp. 546–552.
[23] B. Chen, J. Li, G. Lu, H. Yu, and D. Zhang, “Label co-occurrence
learning with graph convolutional networks for multi-label chest X-ray
image classification,” IEEE J. Biomed. Health Inform., vol. 24, no. 8,
pp. 2292–2302, Aug. 2020.
[24] B. Chen, Z. Zhang, Y. Lu, F. Chen, G. Lu, and D. Zhang, “Semantic-
interactive graph convolutional network for multilabel image recognition,”
IEEE Trans. Syst., Man, Cybern. Syst., vol. 52, no. 8, pp. 4887–4899,
Aug. 2022.
[25] J. L. Elman, “Learning and development in neural networks: The impor-
tance of starting small,” Cognition, vol. 48, no. 1, pp. 71–99, 1993.
[26] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learn-
ing,” in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 41–48.
[27] X. Wang, Y. Chen, and W. Zhu, “A survey on curriculum learning,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 9, pp. 4555–4576,
Sep. 2022.
[28] J. Wei et al., “Learn like a pathologist: Curriculum learning by annotator
agreement for histopathology image classification,” in Proc. IEEE/CVF
Winter Conf. Appl. Comput. Vis., 2021, pp. 2473–2483.
[29] M. Yang, Z. Xie, Z. Wang, Y. Yuan, and J. Zhang, “Su-MICL: Severity-
guided multiple instance curriculum learning for histopathology image
interpretable classification,” IEEE Trans. Med. Imag., vol. 41, no. 12,
pp. 3533–3543, Dec. 2022.
[30] R. Zhao, X. Chen, Z. Chen, and S. Li, “EGDCL: An adaptive curriculum
learning framework for unbiased glaucoma diagnosis,” in Proc. Eur. Conf.
Comput. Vis., Springer, 2020, pp. 190–205.
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY JALANDAR. Downloaded on February 23,2024 at 13:16:49 UTC from IEEE Xplore. Restrictions apply.
[31] J. Yang et al., “Self-paced balance learning for clinical skin disease
recognition,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 8,
pp. 2832–2846, Aug. 2020.
[32] Q. Zhu, N. Yuan, J. Huang, X. Hao, and D. Zhang, “Multi-modal AD
classification via self-paced latent correlation analysis,” Neurocomputing,
vol. 355, pp. 143–154, 2019.
[33] P. Morerio, J. Cavazza, R. Volpi, R. Vidal, and V. Murino, “Curriculum
dropout,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 3544–3552.
[34] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive grow-
ing of GANs for improved quality, stability, and variation,” 2017,
arXiv:1710.10196.
[35] M. De Lange et al., “A continual learning survey: Defying forgetting in
classification tasks,” in IEEE Trans. Pattern Anal. Mach. Intell., vol. 44,
no. 7, pp. 3366–3385, Jul. 2022.
[36] W. Ou, S. Xiao, C. Zhu, W. Han, and Q. Zhang, “An overview of
brain-like computing: Architecture, applications, and future trends,” Front.
Neurorobot., vol. 16, 2022, Art. no. 1041108.
[37] S. Ghosh-Dastidar and H. Adeli, “Spiking neural networks,” Int. J. Neural
Syst., vol. 19, no. 4, pp. 295–308, 2009.
[38] Q. Zhou, C. Ren, and S. Qi, “An imbalanced R-STDP learning rule in
spiking neural networks for medical image classification,” IEEE Access,
vol. 8, pp. 224162–224177, 2020.
[39] A. Garain, A. Basu, F. Giampaolo, J. D. Velasquez, and R. Sarkar,
“Detection of COVID-19 from CT scan images: A spiking neural
network-based approach,” Neural Comput. Appl., vol. 33, no. 19,
pp. 12591–12604, 2021.
[40] A. Paul et al., “Generalized zero-shot chest X-ray diagnosis
through trait-guided multi-view semantic embedding with self-
training,” IEEE Trans. Med. Imag., vol. 40, no. 10, pp. 2642–2655,
Oct. 2021.
[41] Y. Bengio, “Evolving culture versus local minima,” in Growing Adaptive
Machines: Combining Development and Learning in Artificial Neural
Networks. Berlin, Germany: Springer, 2014, pp. 109–138.
[42] T. Gong, Q. Zhao, D. Meng, and Z. Xu, “Why curriculum learning &
self-paced learning work in big/noisy data: A theoretical perspective,” Big
Data Inf. Analytics, vol. 1, no. 1, pp. 111–127, 2016.
[43] P. P. Team, J. K. Gohagan, P. C. Prorok, R. B. Hayes, and B.-S.
Kramer, “The prostate, lung, colorectal and ovarian (PLCO) cancer
screening trial of the national cancer institute: History, organization,
and status,” Controlled Clin. Trials, vol. 21, no. 6, pp. 251S–272S,
2000.
[44] J. Irvin et al., “CheXpert: A large chest radiograph dataset with uncertainty
labels and expert comparison,” in Proc. AAAI Conf. Artif. Intell., 2019,
pp. 590–597.
[45] L. Oakden-Rayner, “Exploring large-scale public medical image datasets,”
Academic Radiol., vol. 27, no. 1, pp. 106–112, 2020.
[46] X. Sun and W. Xu, “Fast implementation of DeLong’s algorithm for
comparing the areas under correlated receiver operating characteristic
curves,” IEEE Signal Process. Lett., vol. 21, no. 11, pp. 1389–1393,
Nov. 2014.
[47] L. N. Smith, “Cyclical learning rates for training neural networks,” in Proc.
IEEE Winter Conf. Appl. Comput. Vis., 2017, pp. 464–472.
[48] C. Yan, J. Yao, R. Li, Z. Xu, and J. Huang, “Weakly supervised deep
learning for thoracic disease classification and localization on chest X-
rays,” in Proc. ACM Int. Conf. Bioinf., Comput. Biol., Health Inform.,
2018, pp. 103–110.
[49] X. Zhu and Q. Feng, “MVC-Net: Multi-view chest radiograph classifica-
tion network with deep fusion,” in Proc. IEEE 18th Int. Symp. Biomed.
Imag., 2021, pp. 554–558.
[50] W. Xu, W. Liu, X. Huang, J. Yang, and S. Qiu, “Multi-modal self-paced
learning for image classification,” Neurocomputing, vol. 309, pp. 134–144,
2018.
[51] N. Sarafianos, T. Giannakopoulos, C. Nikou, and I. A. Kakadiaris, “Cur-
riculum learning of visual attribute clusters for multi-task classification,”
Pattern Recognit., vol. 80, pp. 94–108, 2018.
[52] S. Ghamizi, M. Cordy, M. Papadakis, and Y. L. Traon, “On evaluating
adversarial robustness of chest X-ray classification: Pitfalls and best prac-
tices,” 2022, arXiv:2212.08130.
[53] M. Xu, T. Zhang, Z. Li, M. Liu, and D. Zhang, “Towards evaluating the
robustness of deep diagnostic models by adversarial attack,” Med. Image
Anal., vol. 69, 2021, Art. no. 101977.