2118 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 42, NO.
7, JULY 2023
A New Framework of Swarm Learning
Consolidating Knowledge From Multi-Center
Non-IID Data for Medical Image Segmentation
Zheyao Gao , Fuping Wu , Weiguo Gao, and Xiahai Zhuang
Abstract — Large training datasets are important for deep
learning-based methods. For medical image segmentation,
it could be however difficult to obtain large number of
labeled training images solely from one center. Distributed
learning, such as swarm learning, has the potential to use
multi-center data without breaching data privacy. However,
data distributions across centers can vary a lot due to the
diverse imaging protocols and vendors (known as feature
skew). Also, the regions of interest to be segmented could
be different, leading to inhomogeneous label distributions
(referred to as label skew). With such non-independently
and identically distributed (Non-IID) data, the distributed
learning could result in degraded models. In this work,
we propose a novel swarm learning approach, which assem-
bles local knowledge from each center while at the same
time overcomes forgetting of global knowledge during local
training. Specifically, the approach first leverages a label
skew-awared loss to preserve the global label knowledge,
and then aligns local feature distributions to consolidate
global knowledge against local feature skew. We vali-
dated our method in three Non-IID scenarios using four
public datasets, including the Multi-Centre, Multi-Vendor
and Multi-Disease Cardiac Segmentation (M&Ms) dataset,
the Federated Tumor Segmentation (FeTS) dataset, the
Multi-Modality Whole Heart Segmentation (MMWHS) dataset
and the Multi-Site Prostate T2-weighted MRI segmenta-
tion (MSProsMRI) dataset. Results show that our method
could achieve superior performance over existing methods.
Code will be released via https://zmiclab.github.io/
projects.html once the paper gets accepted.
Index Terms — Medical image, non-IID, segmentation,
swarm learning.
I. I NTRODUCTION
T HE gains of deep learning for medical image segmenta-
tion could highly depend on the amount and diversity
Manuscript received 5 August 2022; revised 13 October 2022;
accepted 5 November 2022. Date of publication 9 November 2022;
date of current version 29 June 2023. This work was supported by the
National Natural Science Foundation of China under Grant 61971142,
Grant 62111530195, Grant 62011540404, and Grant 71991471.
(Corresponding author: Xiahai Zhuang.)
Zheyao Gao and Xiahai Zhuang are with the School of Data
Science, Fudan University, Shanghai 200433, China (e-mail:
zygao20@fudan.edu.cn; zxh@fudan.edu.cn).
Fuping Wu is with the School of Data Science, and Depart-
ment of Statistics, Fudan University, Shanghai 200433, China (e-mail:
17110690006@fudan.edu.cn).
Weiguo Gao is with the School of Data Science, and School of
Mathematical Sciences, Fudan University, Shanghai 200433, China
(e-mail: wggao@fudan.edu.cn).
Digital Object Identifier 10.1109/TMI.2022.3220750
1558-254X © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
of labeled training data. The quantity of data at a single
medical center is usually limited, especially for rare dis-
eases [1]. Centralized learning methods, which assemble data
from multiple centers, may not be applicable due to the data
privacy issue [2]. To solve this problem, federated learning has
been proposed, which trains a model on distributed datasets
without the exchange of privacy-sensitive data between
centers [3].
Recently, research on federated learning has gone beyond
privacy issues, to further investigate novel methods well
handling issues of security, transparency and fairness [4].
For healthcare applications [2], there are two widely adopted
approaches. One is to rely on an aggregation server on secure
hardware [5], [6], and the other is to adopt the peer-to-peer
communication [4], [7]. Among them, swarm learning, a new
decentralized paradigm, has been recently proposed [4]. This
paradigm can keep both the manipulation of data and parame-
ters locally. The decentralized paradigm provides a promising
solution for privacy protection and fairness. However, the
performance could degrade when it comes to the problem
of non-independent and identically distributed (Non-IID) data
coming from different centers [8], [9]. This is mainly due to
the fact that the local training on Non-IID data could update
the local models in different directions, and averaging these
models with large discrepancy may deviate the optimization
and deteriorate performance of the final model.
The main challenges in Non-IID data for decentralized
learning have three folds, i.e., feature skew, label skew and
quantity skew [9]. The former two are dominant and will be
fully investigated in this work, while the third one is less
studied here, as it becomes an issue of sample sizes for image
segmentation tasks. Fig. 1 illustrates the two skew issues with
segmentation of short-axis cardiac magnetic resonance (CMR)
images. Feature skew could originate from the difference in
imaging protocols, the strength of magnetic field in magnetic
resonance imaging (MRI), or different demographics, which
lead to the covariate shift [11].
Label skew commonly exists in multi-center data with
diverse forms, among which one prevailing scenario is the
partial annotation of training images. Fig. 1 shows three CMR
images with three types of annotations for three different stud-
ies, i.e., different structures of interest to be segmented. This
is because cardiac MRI could be used in studies of various
cardiac disorders, and the images from different centers could
 GAO et al.: NEW FRAMEWORK OF SWARM LEARNING CONSOLIDATING KNOWLEDGE FROM MULTI-CENTER NON-IID DATA
Fig. 1. Illustration of feature skew and label skew. The first row shows
the log histogram of typical images from three centers. The sample slices
of different distributions are presented in the second row. The third row
presents three types of partial labeling and the weak label form [10],
where the four elements in label vectors denote the probabilities of pixels
being non-cardiac background (BG), myocardium (Myo), left ventricle
(LV) and right ventricle (RV), respectively. UN means unlabeled pixels.
be solely partially annotated, due to the diverse interests of
clinical studies. For example, in myocardial viability studies
only the myocardium of left ventricle (LV) is annotated [12];
while for the diagnosis of right ventricle (RV) related patholo-
gies (e.g. pulmonary hypertension, coronary heart disease and
dysplasia), only the right ventricle requires segmentation [13].
Therefore, given the same type of images, the segmentation
could be conceptually different. These inconsistent annotations
from the same imaging area across centers result in the
Non-IID distribution of the labels, which is referred to as the
label skew.
To alleviate the skew problems, approaches have been
proposed [1]. For example, Shoham et al. [14] proposed
to adopt the elastic weight consolidation strategy (EWC) to
prevent the local models from drifting apart by overcoming the
catastrophic forgetting. Li et al. [15] proposed a personalized
method that keeps the batch normalization locally during
model aggregation to mitigate the feature skew problem.
Although these methods have shown potential for classification
tasks when training with Non-IID data, they do not take into
consideration the label skew in segmentation tasks. This could
lead to different local updating directions, and thus could
degrade performance of the aggregated model.
In this work, we propose a new privacy protected, decen-
tralized distributed learning framework for medical image
segmentation, which tackles both of the feature and label skew
problems. Specifically, we propose the label skew-awared
(LaSA) loss with unbiased probability for unlabeled structures,
which maximizes the prediction for the most probable class
predicted by the global model during local training. This LaSA
loss is able to preserve the full label knowledge for local
models. Furthermore, we propose the feature skew-awared
(FeSA) regularization and use batch normalization (BN) statis-
tic matching to align the feature distributions between centers,
which alleviates the effect of feature skew. We validate our
proposed framework comprehensively in three scenarios using
four public datasets, and the results demonstrate the great
potential in the tasks of medical image segmentation.
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
2119
Our contributions are summarized as follows:
• We propose a novel decentralized distributed learning
framework for medical image segmentation, which could
train models using partially labeled images from multiple
centers for full label segmentation.
• We introduce an unbiased probability-based, label
skew-awared loss to handle label skew, by consolidating
global knowledge from local knowledge of partial labels.
• We propose a feature statistic-based regularization term
that could cope with feature skewed data by distilling
robust feature distribution knowledge during local train-
ing.
• We validate the proposed framework in three segmen-
tation tasks to demonstrate its applicability in different
scenarios.
This paper is organized as follows: Section II provides a
brief summary of related works, including federated learning,
swarm learning and distributed learning with Non-IID data.
In Section III, we first present the standard swarm learning as
preliminaries, and then describe the framework of our method
in details. The results of parameter studies and comparisons
are presented in Section IV. Finally, Section V concludes the
paper.
II. R ELATED W ORKS
A. Data Privacy and Distributed Learning
The concept of distributed machine learning was first
introduced by [3]. They proposed the first federated learn-
ing algorithm, i.e., FedAvg, that allows multiple clients to
collaboratively train a deep learning model, while keeping
the data stored locally. Although FedAvg meets the basic
requirements for data privacy protection, it needs to send
user-sensitive data (i.e., model parameters or gradients) to
a central server. Since private information could be recov-
ered from the model gradients or parameters during training,
the previous works [16], [17] have proposed secured meth-
ods for parameter aggregation. In medical images analysis,
Li et al. [17] proposed to add random noise to shared weights
to avoid the leaking of private information. Most recently,
swarm learning [4] was proposed to ensure the security and
fairness by sending the parameters peer-to-peer through an
encrypted and decentralized manner of blockchain. Compared
with other privacy protection methods, it is effective and
efficient as it only requires minimal modification to the basic
FedAvg method. However, swarm learning also suffers from
the same problem when learning from Non-IID data as FedAvg
does, which we mainly aim to tackle in this work.
B. Distributed Learning With Non-IID Data
Since the data distribution is different in each center, a com-
mon idea is to build a personalized model from the shared
global model based on local distributions. A straightforward
method is to fine-tune the global model on local data [18].
However, such naive fine-tuning could result in overfitting
on local training data [9]. To avoid overfitting, some meth-
ods [19], [20] proposed to build a personalized model, which
 2120
TABLE I
R EFERENCES FOR THE M ATHEMATICAL S YMBOLS
was trained with local data and applied distillation strategies.
Other methods [15], [21] proposed to keep distribution related
layers in local centers during model aggregation, instead of
building an entire personalized model. Although the methods
based on personalization have achieved excellent performance
in Non-IID scenarios, it is arduous to yield a robust global
model. Hence, the generalization ability of the resulting mod-
els to unseen data from other centers could be still limited.
Another strategy to handle this problem is to add regulariza-
tion terms during local training to avoid divergence of local
models. For example, Li et al. [8] first proposed to use an
L2-norm regularization to constrain parameters from deviating
too far from the global model. To design a more delicate
term, Shoham et al. [14] proposed to adapt a solution tackling
catastrophic forgetting by estimating the importance matrices
for all parameters.
C. Overcoming Forgetting
The concept of overcoming forgetting was first introduced
in continual learning by [22], which is aimed to maintain the
performance of a model for old tasks when the model is being
trained for new tasks [23]. It was then applied in distributed
learning to handle the Non-IID issue by [14]. Here, we mainly
focus on the regularization-based approaches, which could be
easily adapted to the tasks of distributed learning. Inspired by
the work of [22], several works have been proposed to restrict
the alteration of parameters that are important to previous
tasks [24], [25]. However, the estimation of importance matri-
ces could be inaccurate, thus undermining the effectiveness
of the techniques. Alternatively, there are methods alleviating
the forgetting through feature distillation. Douillard et al. [26]
proposed to use spatially pooled features for the preservation
of old knowledge. Similarly, Arthur et al. [27] applied this
idea in continual segmentation tasks. In feature distillation
methods, the strategy for feature extraction is essential to the
model performance. Based on this idea, we propose to employ
distribution-related features to deal with the Non-IID issues.
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 42, NO. 7, JULY 2023
III. M ETHOD
A. Preliminaries
Let Pk (X , Y) be the joint distribution of image variable
(X ) and label variable (Y) in center k, where k = 1 . . . K ;
let P(X , Y) be the joint distribution for all data. The goal
of distributed learning is to train a model, denoted as f θ :
X → Y, with parameter set θ , by minimizing the risk on
data from the true distribution without data exchange between
centers. In this work, we solely consider a degraded objective
that minimizes summation of empirical risks on each local
distribution, i.e.,
θ̂ = arg min E(x,y)∼P(X ,Y )[( f θ (x), y)] (1)
θ

K
≈ arg min E(x,y)∼Pk (X ,Y )[( f θ (x), y)], (2)
θ k=1
where, (·) is the loss function measuring the dissimilarity
between prediction and ground truth. Table I summarizes the
key symbols.
Swarm learning involves local update and communication
between centers [4]. At each round of communication (denoted
with superscript t), every center receives the model parameters
from the other K − 1 centers and then performs parameter
aggregation,

K
nk
θ [t ] = θk[t ] , (3)
N
k=1
where n k isthe number of training samples from center k
K
and N = k=1 n k . Each local model is initialized by the
aggregated parameters, and then updated via local training
until the next communication round. The objective of local
training for center k is formulated as follows,
min E(x,y)∼Pk (X ,Y )[( f θk (x), y)], (4)
θk
which leads to local optimal θk[t ] . This process is repeated
until the aggregated model converges. This framework is
similar to the most popular federated averaging algorithm
(FedAvg) [3], except that the aggregated model is decen-
tralized, namely there is no central server, and the commu-
nication is performed through blockchain-based peer-to-peer
networking [28] between all clients. However, the underly-
ing assumption of vanilla swarm learning is that local data
from each center should be in the same distribution, i.e.,
Pk (X , Y) = P(X , Y). In reality, this assumption could be
false, especially in the scenario of multi-center medical image
analysis, where data are Non-IID.
B. Non-IID Swarm Learning
In this work, we consider a Non-IID scenario where both
feature skew and label skew can exist. Specifically, we focus
on the covariate shift for the problem of feature skew (dif-
ferent P(X )) and the concept shift for label skew (different
P(Y|X )). The former can be caused by different protocols
of image acquisition, and the latter is common in medical
image segmentation, as different centers could have different
 GAO et al.: NEW FRAMEWORK OF SWARM LEARNING CONSOLIDATING KNOWLEDGE FROM MULTI-CENTER NON-IID DATA 2121

Fig. 2. The overall framework of the proposed method: the paradigm of swarm learning (left) and the proposed method for local training (right).
In local training, the proposed method adopts LaSA loss and FeSA regularization to tackle the problems from label and feature skew, respectively.
As shown in the procedure E, LaSA is computed with unbiased probability and unbiased partial label. The unbiased probability is calculated by
summing up the probabilities for unlabeled classes and the unbiased partial label is derived similarly by summing up values for unlabeled classes
in the weak label form of partial label. RFeSA is computed by matching the sample mean and variance from local model with the running mean and
variance parameters in batch normalization layers of the global model.

structures of interests to segment, i.e., partial labeling. Without
confusion, we denote the partial label from center k as yk .
Both label skew and feature skew can diverge a standard
local training from the global updates, leading to degraded per-
formance of the aggregated model [8]. To tackle this, we pro-
pose a novel decentralized learning framework. As shown in
Fig. 2, the left subfigure presents the paradigm of standard
swarm learning, as described in Section III-A. Each center
sends the local model to a decentralized communication net-
work and receives models from other centers to derive the
global model. Our contribution is focused on the local training
process which is illustrated in the right part of Fig. 2. Our
method uses two new functions, i.e., a label skew-awared
(LaSA) loss and a feature skew-awared (FeSA) regularization
term. LaSA loss adopts the unbiased probability derived from
the output of networks to formulate the segmentation loss for
label skew scenario. FeSA regularization handles the feature
skew problem by matching the statistic of intermediate features
in local clients with the global model. Formally, the objective
for the local training with partially labeled data (x k , yk ) is
given by,
( f θk (x k ), yk ) =  La S A ( f θk (x k ), yk ) + λR F eS A (θk ; θ [t ] , x k ),
(5)
where λ is the balancing parameter. The pseudo code for the
whole pipeline is shown in Algorithm 1.
The proposed strategies deal with the label skew and feature
skew challenges in distributed learning by avoiding the diver-
gence of local updates during local training. This divergence
resembles the consequence of catastrophic forgetting in con-
tinual learning [23]. In local training, each center fine-tunes
the aggregated model using local data without an access to the
data from other centers. Hence, the local model could forget
the knowledge learned from the segmentation tasks in other
centers, resulting in different updating directions.
Therefore, we first develop the LaSA loss to preserve the
global knowledge of full label (global distribution) when train-
ing the model locally. Then, we design the FeSA regularization
to distill the knowledge of feature distributions from the
global model. In the following, we elaborate on the details
of the two contributions in Section III-C and Section III-D,
respectively.
C. Label Skew-Awared Loss LaSA
For pixel i , let q i , yki and y i respectively denote the
segmentation prediction vector, the gold standard label vector
of partial annotations from center k and the ground truth label
which is a one-hot vector. Elements of these three vectors can
be accessed using index of c, which is also the index of the
class set. Let C be the set of all classes in the segmentation
task. In label skew scenario, Ck represents the set of annotated
classes in center k, and C¯k = C\Ck denotes the set of
unlabeled classes.
For partial label scenario, we use the weak label form [10]
as Fig. 1 illustrates. Formally, for a pixel i of an image x
which is partially annotated with label class set Ck in center
k, if pixel i is annotated, we have yki = y i ; otherwise, the
elements of vector yki are given by,

0, for c ∈ Ck
yk,c =
i
.
1
|C̄ |
, for c ∈ C̄k
k
For the label skew issue, we propose a new loss to preserve
the label knowledge for unlabeled classes, i.e., LaSA loss. The

Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
 2122 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 42, NO. 7, JULY 2023

Algorithm 1 Non-IID Swarm Learning

Notations: Initialize model parameters: θk[0] for each
center k; total optimization epochs: T ; running mean and
variance in the lth layer of global model: μl , σl2 ; sample
mean and variance from the lth layer of local model:
μ̃k,l , σ̃k,l
2 .

for each epoch t = 0, 1, 2, . . . , T − 1 do

for each center k = 1, 2 . . . , K do
qk , μ̃k , σ̃k2 = f θk (x k )
compute  q̂k , ŷk using Eq. (7), (8) Fig. 3. Illustration for the gradients of (a) CE, (b) PCE and (c) LaSA loss
on an unlabeled pixel in one center with regard to the predicted logits.
 La S A = i∈ seg (q̂ki , ŷki ) (Please refer to Eq. (6)) In this center, the labeled class is c2 . As shown by the blue boxes, the
R F eS A = l μl − μ̃k,l 2 + σl2 − σ̃k,l 2  (Please
2 class of this unlabeled pixel is predicted to be c3 by the global model.
refer to Eq. (15)) The dash line illustrates the resulting probability distributions by different
losses. Figure (a) indicates that by using CE , the gradient computed for
 =  La S A + R F eS A the most probable class is the least, resulting more evenly distributed
θk[t +1] ← Opti mi zer (, θk[t ] ) probabilities. Figure (b) shows that PCE does not compute gradients
end for for unlabeled classes, which cannot prevent the knowledge forgetting.
 K nk [t +1] In figure (c), the proposed LaSA not only derives a negative gradient for
θ [t +1] ← k=1 N θk c2 , using the knowledge from local training data, but also generates the
for each client k do largest positive gradient for the most probable class (c3 ) predicted from
θk[t +1] ← θ [t +1] the global model. This consolidates the global knowledge.

end for
end for  i i
2i∈ ŷk,C̄ q̂k,C̄
−  , (11)
i∈ yk,C̄ +
i i
i∈ q̂k,C̄

LaSA loss is formulated with unbiased probability, as follows, where αc and γ are hyperparameters in focal loss, which are
 introduced to control the weights of losses over each class
 La S A( f θk (x k ), yk ) = seg (q̂ki , ŷki ), (6)
and pixel. One can see that the formulation in Eq. (9) is
i∈
similar to the marginal loss in [32]. We will show that the
where  is the set of pixels in the image, q̂ki and ŷki are the label skew-awared loss in our framework could consolidate
unbiased prediction probability and unbiased partial label for label knowledge.
pixel i , respectively, defined as, In the following, we provide the theoretical explanation

qci if c ∈ Ck of the proposed LaSA loss by analyzing the problem of
q̂k,c = 
i
, (7) fully-supervised segmentation losses in label skew scenario.
¯ q i
if c ∈ C¯k
j ∈C k j Without loss of generality, here we take CE loss as an example.
and For segmentation task in local client k, the overall CE loss
 is the aggregation of CE losses over each pixel, which is given
i
yk,c if c ∈ Ck
i
ŷk,c =  . (8) by,
¯
j if c ∈ Ck
i
j ∈C¯k yk,   
Here, we further introduce two symbols for convenience of C E ( f θik (x k ), yki ) = − i
yk,c log(qci ) + i
yk,c log(qci ) .
i
notation, i.e., q̂k, C̄
= q̂k,c
i
, ∀c ∈ C¯k and ŷk, i
C̄
= ŷk,c
i
, ∀c ∈ C¯k , c∈C k c∈C̄ k
for the derivation of losses over unlabeled classes. (12)
In Eq. (6), seg (·) could be adapted from the loss func- One can see that for an unlabeled pixel i , the CE loss
tions for fully-supervised segmentation. Here, we provide the maximizes the output probabilities qci for all unlabeled classes
implementation with cross entropy (CE) loss [29], focal (FC) c ∈ C̄k . However, since log(·) is a concave function, the
loss [30] and Dice (DC) loss [31], as follows, gradient computed for the most probable class is the least.
    
As shown in Fig. 3, C E could result in more evenly dis-
 La S A[C E] = − i
ŷk,c i
log(q̂k,c ) + ŷk,
i
C̄
log( q̂ i
k,C̄
) ,
tributed probabilities for unlabeled classes. Optimization of
i∈ c∈C k
this loss leads to increase of the uncertainty over unlabeled
(9)
    pixels, which expedites the forgetting of label knowledge for
i γ
 La S A[F C] = − αc ŷk,c
i
(1 − q̂k,c ) i
log(q̂k,c ) unlabeled structures.
i∈ c∈C k An alternative solution is to apply partial CE that only
 computes losses over labeled classes [33],
γ
+ ŷk,
i
C̄
(1 − q̂ i
k,C̄
) log( q̂ i
k,C̄
) , 
iPC E ( f θk (x k ), yk ) = − i
yk,c log(qci ). (13)
(10)
 c∈C k
 i q̂ i
2 i∈ ŷk,c k,c
 La S A[DC] = 1 −   In this case, no loss is computed over unlabeled pixels. How-
i∈ yc +
i i
c∈C k i∈ q̂k,c ever the prediction for unlabeled pixels is affected by the losses

Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
 GAO et al.: NEW FRAMEWORK OF SWARM LEARNING CONSOLIDATING KNOWLEDGE FROM MULTI-CENTER NON-IID DATA
derived for labeled pixels. Although it does not explicitly
degrade the performance of a model over the unlabeled pixels,
it cannot alleviate the catastrophic forgetting during local
training.
Compared with the above losses, our proposed LaSA loss,
adopting the unbiased probability, could derive a proper esti-
mation for losses over unlabeled pixels and consolidate the
label knowledge from the global model. In Eq. (9), the loss
for an unlabeled pixel i is derived by the cross entropy between
i E(Xl ) ≈
the unbiased prediction probability q̂k, and unbiased partial
C̄
i
label ŷk,C̄ . In this formulation, the largest gradient is obtained
for the most probable unlabeled class among classes in C̄k . The
derivative of Eq. (9) with respect to the model output could
be derived by,
∂iLa S A[C E]( f θk (x k ), yk ) 1 features from layer l of the local model in center k. The
= −qci ( − 1) ∝ −qci . (14)
∂ f ci i
q̂k, C̄
From this formula, one can see that LaSA tends to maximize
the prediction for the most probable class predicted by the
global model during local training. In other words, the loss is
able to memorize the prediction of global model on unlabeled
pixels during local training, thus consolidates the global
knowledge of full labeling. Fig. 3 illustrates and compares the
gradients computed by LaSA and other loss functions.
D. Feature Skew-Awared Regularization RFeSA
For feature skew, we propose to introduce an extra reg-
ularization term using the feature statistics before the batch
normalization (BN) module [34], i.e., the FeSA regularization
term,
 MMWHS [38] dataset. We first demonstrated the effectiveness
R F eS A (θk ; θ [t ] , x k ) = μl − μ̃k,l 2 + σl2 − σ̃k,l
2
2 ,
l for feature skew in Section IV-C. We then compared the
where, μl and σl are the running mean and variance of
parameters in BN module for layer l in the global model;
l 2 ∈ RC l are the sample mean and variance
μ̃k,l ∈ RC and σ̃k,l
in the local model. The subscript k indicates the variables are
associated with local client k and the C l denotes the number
of channels in layer l. They are calculated along the batch and
spatial dimensions of the features from layer l, given a batch
of input images sampled from local distribution Pk (X ).
In Eq. (15), the running mean and variance (μl and σl )
are calculated through aggregation of the corresponding para-
meters in the local model of last communication epoch. The
computation of the running mean and variance parameters in
local models is the same as in [34], namely they are updated
by,
μ[m] [m−1]
k,l = Mμk,l + (1 − M)μ̃k,l
(σ 2 )[m]
k,l = M(σ 2 )[m−1]
k,l + (1 − M)σ̃k,l
2
where m represents the index of iteration and M is the
momentum between (0, 1). It is set empirically according to
the number of batches in one local training epoch.
The intuition behind the method is that the feature statistics
computed in BN layers contain the traits of image distribution
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
2123
P(X ) [35]. The feature statistics learned for local distribu-
tion Pk (X ) could diverge from those of the global model.
To preserve the knowledge of global statistics of layer l (i.e.
E(Xl ) and D(Xl )), we consider to force the sample statistics
calculated during local training to be close to the running
statistics in global model using L2 distance. Here, we estimate
the true global feature statistics by,
1 
K
nk 2 
K
nk
E k (Xl ) ≈ μk,l = μl , (18)
N N
k=1 k=1
K K
1 nk 2 nk 2
D(Xl ) ≈ Dk (Xl ) ≈ σ = σl2 , (19)
N N k,l
k=1 k=1
where E k (Xl ) and Dk (Xl ) are the mean and variance of
second approximations in both formulas are based on the
assumption that the running statistics are accurate estimations
of the true statistics. In Eq. (18), the weighted sum of local
means is an unbiased estimation for the mean of global
feature distribution which supports the first approximation.
For variance estimation, the first approximation in Eq. (19)
is based on the assumption that the local mean μk,l is similar
for each center. This assumption is reasonable because we
attempt to align the local means with the global mean during
training.
IV. E XPERIMENTS
We validated the applicability of the proposed method
using four datasets, i.e., the M&Ms dataset [36], the
FeTS dataset [37], the MSProsMRI [11] dataset and the
of the LaSA loss for label skew and FeSA regularization
(15) framework with other state-of-the-art (SOTA) methods in three
situations in Section IV-D, including the scenario having
both label skew and feature skew issues, the scenario with
solely label skew and the scenario with solely feature skew.
In the experiments, we performed Wilcoxon test to report the
significance of the difference between two approaches.
A. Datasets
1) M&Ms Dataset: is composed of 320 short-axis Cardiac
Magnetic Resonance (CMR) cases from 4 different scan-
ner vendors, including 95 cases from Siemens (Vendor A),
125 cases from Philips (Vendor B), 50 cases from General
Electric (Vendor C) and 50 cases from Cannon (Vendor D).
It provides annotations for 3 cardiac structures, including left
ventricle (LV), right ventricle (RV) and the left ventricular
(16) myocardium (Myo). We resampled all data with the in-plane
resolution of 1.25×1.25 mm and cropped them into 192×192
(17)
region of interest (ROI). We took each vendor as a local client
such that the image distribution in each client is different.
To simulate the label skew, during training, only the anno-
tations of LV and RV are available for the data in Vendor
A, LV and Myo for Vendor B, all cardiac structures as one
foreground (illustrated in Fig. 1) for Vendor C and RV for
 2124
TABLE II
A BLATION S TUDY FOR THE P ROPOSED M ETHOD. R ESULTS A RE E VALUATED IN D ICE S CORE
Fig. 4. Visualization of segmentation results on M&Ms dataset. Arrows highlight the areas which are referred to in the text.
TABLE III
S UMMARY OF M&M S D ATASET. O NLY PART OF L ABELS W ERE U SED
FOR T RAINING IN E ACH C ENTER . LV: L EFT V ENTRICLE , RV: R IGHT
V ENTRICLE ; M YO : M YOCARDIUM ; BG: B ACKGROUND
Vendor D. We randomly split the data of each client by the
ratio of 50%:15%:35% for training, validation and test. The
dataset information is summarized in Table III.
2) FeTS Dataset: includes multi-institutional multi-
parametric Magnetic Resonance Imaging (mpMRI) scans of
brain tumor from Federated Tumor Segmentation (FeTS)
challenge 2022 [37]. It was built upon the dataset from
RSNA-ASNR-MICCAI BraTS 2021 challenge [39] with their
real-world partitioning and the collaborative of independent
institutions in a real-world federation [40]. The training
data consists of 1251 cases and each case has four 240 ×
240 × 155 structural MRI images including native (T1),
post-contrast T1-weighted (T1Gd), T2-weighted (T2), and
T2 FLuid Attenuated Inversion Recovery (FLAIR) volumes.
The pre-processing steps for all images include z-score
normalization, rigid registration and resolution resampling
to 1mm 3 . The provided annotation includes the enhancing
tumor (ET), necrotic tumor core (NCR) and peritumoral
edematous and infiltrated tissue (ED). To simulate the label
skew scenario, we removed the annotations for one or two
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 42, NO. 7, JULY 2023
structures in the training data of each center. Since quantity
skew is not the main focus of this work, we merged the
data from the centers listed in the partitioning-2 file from
the FeTS challenge. The information of the final partitioned
dataset is summarized in Table IV. In the experiment, the
dataset was split by the ratio of 60%:15%:25% for training,
validation and test in each center.
3) MMWHS Dataset: has 60 cardiac CT images from MIC-
CAI’17 Multi-Modality Whole Heart Segmentation challenge.
The annotations for the whole heart substructures include the
left ventricle (LV), right ventricle (RV), left atrium (LA),
right atrium (RA), myocardium (Myo), ascending aorta (AO),
and the pulmonary artery (PA). We resampled the images
into 2×2×2 mm and cropped them with ROI of 96×96×64.
To simulate the scenario with label skew, we first randomly
and equally divided them into 4 parts as the datasets for each
center. We performed 4-fold cross validation on this dataset
in our experiment. In each center, we then only used the
annotations of 2 or 3 substructures in each center during
training. Specifically, we have Myo, LV and LA for Center
A; RA and PA for Center B; RV, RA and AO for Center C;
RV, AO and PA for Center D. The dataset information is
summarized in Table V.
The MSProsMRI dataset includes 30 cases from
RUNMC [41](Center A), 30 cases from BMC [41](Center
B), 19 cases from HCRUDB [42](Center C), 13 cases from
UCL [43](Center D), 12 cases from BIDMC [43](Center E),
and 12 from HK [43](Center F). We resampled all these
data with the axial plane resolution of 0.625×0.625 mm and
cropped them into ROI of 384×384. We performed 4-fold
 GAO et al.: NEW FRAMEWORK OF SWARM LEARNING CONSOLIDATING KNOWLEDGE FROM MULTI-CENTER NON-IID DATA 2125

TABLE IV
S UMMARY OF F E TS D ATASET. T HE D ATASET WAS M ANUALLY S PLIT I NTO N INE C ENTERS . O RG C ENTER D ENOTES C ENTER I NDEX OF THE
O RIGINAL PARTITION P ROVIDED BY THE F E TS C HALLENGE . O NLY PART OF L ABELS W ERE U SED FOR T RAINING IN E ACH C ENTER . ET:
E NHANCING T UMOR ; NCR: N ECROTIC T UMOR C ORE ; ED: P ERITUMORAL E DEMATOUS AND I NFILTRATED T ISSUE ; BG: B ACKGROUND

TABLE V
S UMMARY OF MMWHS D ATASET. T HE D ATASET WAS M ANUALLY
S PLIT IN F OUR C ENTERS . O NLY PART OF L ABELS W ERE U SED FOR
T RAINING IN E ACH C ENTER . LV: L EFT V ENTRICLE ; RV: R IGHT
V ENTRICLE ; LA: L EFT ATRIUM ; RA: R IGHT ATRIUM ; M YO :
M YOCARDIUM OF LV; AO: A SCENDING AORTA ; PA: P ULMONARY
A RTERY

TABLE VI Fig. 5. Plot of the knowledge forgetting in Dice score during epoch 200 in
L AYER C HOICE FOR BN S TATISTIC M ATCHING IN F E SA Center D. The vertical axis represents the average Dice score for LV and
R EGULARIZATION . C RITICAL L AYERS R EFER TO THE D OWNSAMPLE Myo achieved by local model in center D on the validation data from all
AND ( OR ) U PSAMPLE C ONVOLUTION L AYERS other centers.

using M&Ms dataset. Swarm+FeSA method was implemented

cross validation on this dataset in our experiments. Since the
annotations only include one structure, we assume no label
skew in this setting. As described in the work of [11], the
feature distribution of the dataset varies across these centers,
causing significant feature skew across centers.
B. Implementation Details
To avoid the effect of quantity skew, we randomly sampled
equal amount of images in each center with random aug-
mentations including rotation, flipping and elastic deformation
during one local training epoch. All images were normal-
ized using z-score normalization before data augmentation.
For M&Ms and MSProsMRI datasets, we adopted the 2D
UNet [44] as backbone, and trained the networks using Adam
optimizer with the initial learning rate of 1e-3, batch size of
128 and 15 iterations per epoch. For both FeTS and MMWHS
datasets, we referred to nnUnet [45] and designed a 3D
segmentation network. The initial learning rate was set to 2e-
4. Random crop was performed during training. The batch
size, number of cases sampled during one epoch and patch
size were 2, 120, 128 × 128×128 for FeTS datasets and 6,
30, 96 × 96×64 for MMWHS datasets. For all experiments,
we empirically set the λ in Eq.(5) to be 0.1, the momentum
M in Eq. (17) to be 0.9 and trained the networks with
500 epochs. The label skew-awared loss was implemented with
cross entropy. The framework was implemented using Pytorch
and ran on one Nvidia RTX 3090 GPU.
C. Parameter Study
1) Effectiveness of LaSA Loss and FeSA Regularization: We
studied the effectiveness of LaSA loss and FeSA regularization
by adding the FeSA regularization to the partial cross entropy
(PCE) loss (Eq. (13)) in the local training of swarm learning.
Swarm+LaSA method replaced the PCE loss with LaSA loss
but without regularization. According to Table II, by applying
regularization term, the improvement from the swarm learning
method was marginal. However, when replacing the naive
loss with LaSA loss, the average Dice score was improved
by 7.3%. This is probably due to the fact that label skew
could trigger the forgetting of label knowledge in local model,
which is more likely to diverge the local training from the
global update than feature skew. One can also observe from
Table II that our proposed method, employing both the regu-
larization term and LaSA loss obtained statistically significant
improvement (Wilcoxon test p-value < 0.05) of 2.3% Dice
score, compared to the method with solely LaSA loss, i.e.,
Swarm+LaSA. This is reasonable because once the label skew
is tackled, the feature skew would become the main issue that
affects the divergence between the local training and the global
update.
The qualitative segmentation examples are shown in Fig. 4
for visual comparisons. One can observe that combining
the two terms could produce more accurate results, and the
improvement is particularly evident in the challenging struc-
ture, i.e., RV.
To further validate that the proposed method could alleviate
the catastrophic forgetting during local training, we visualized
the global knowledge preserved by local model during train-
ing. Here, we illustrated the knowledge preservation in Center
D. The global knowledge was measured by the average Dice
score of unlabeled structures (i.e., LV and Myo) achieved by
the local model on validation data from Center A, B and C.
According to Fig. 5, adopting the proposed training strategy,
the performance hardly declined on data from other centers,

Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
 2126 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 42, NO. 7, JULY 2023

TABLE VII
C OMPARISON S TUDY ON M&M S D ATASETS . R ESULTS A RE E VALUATED IN D ICE S CORE . B OLD T EXT D ENOTES THE B EST R ESULTS ACHIEVED BY
D ISTRIBUTED L EARNING M ETHODS W ITH PARTIAL L ABEL

TABLE VIII
C OMPARISONS OF D IFFERENT S TRATEGIES FOR L ABEL S KEW ON MMWHS D ATASETS . R ESULTS A RE E VALUATED BY 4-F OLD C ROSS
VALIDATION IN D ICE S CORE . B OLD T EXT D ENOTES THE B EST R ESULTS ACHIEVED BY D ISTRIBUTED L EARNING M ETHODS W ITH PARTIAL L ABEL

every training iteration, then the distributed learning is equiv-

alent to the centralized training method with partial label.
Here we studied how communication rate could influence
the performance of the aggregated model. We conducted the
ablation study with model aggregation per 5, 15, 30, 50, 100,
200, 500 iterations. As shown in Fig. 6, the performance was
robust with solely 0.5% discrepancy in Dice if the commu-
nicate rate was set between 5 and 100 iterations. Obvious
decrease was only observed for the communicate rate larger
Fig. 6. Effect of the communication rate on the performance of M&Ms than 200 iterations.
datasets.

while the performance of the swarm learning method showed
dramatic drop. It means that the proposed regularization and
LaSA loss could effectively consolidate old knowledge, thus
keeping the local training consistent with global updates.
2) Choice of Layers for FeSA Regularization: Because using
all the layers for BN statistic matching could restrict the
plasticity of the model, we conducted experiments to select
a better layer subset for the computation of the regularization
term, i.e., R F eS A (θk ; θ [t ] , x k ) in Eq. (15). Since the model we
adopted was UNet, we studied whether the features in encoder
layers or decoder layers play a more important role in the
regularization. We first had all the layers in the encoder or
decoder for regularization. According to Table VI, we found
that there was no significant difference across these choices
(Wilcoxon test p-values > 0.6 for each pair of choice).
Hence, we further relaxed the constraint, and chose solely
features from several critical layers (i.e. features after the each
downsample or upsample convolution layer) for BN statistic
matching. The results in Table VI showed that using the
features from upsample layers for regularization could obtain
better results than using features from downsample layers.
3) Effect of Communication Rate: In swarm learning, com-
munication rate refers to the number of epochs or iterations
in local training between two consecutive model aggregation
operations. Ideally, if the model aggregation is implemented
D. Comparison Study
In this section, we performed a comprehensive comparison
study with other SOTA methods. The methods chosen for
comparisons should have the following three features: (1)
distributed learning methods related to the idea of knowledge
consolidation; (2) global-model-based methods, in which all
centers share one model that is generalizable for data from
unseen centers; (3) methods based on decentralized frame-
works without server-side manipulation.
1) Application to Multi-Center and Multi-Vendor Cardiac Seg-
mentation With Label and Feature Skews: To further validate
the proposed method in the scenario with both label and
feature skew, we compared it with other SOTA methods for
cardiac segmentation on M&M dataset, including: (1) Fed-
Prox [8], which used L2 regularization during local training;
(2) FedCurv [14], which used importance matrix to penalize
the change of important parameters; (3) FML [19], which used
personalization strategy; (4) Swarm, which was the standard
swarm learning framework using partial cross-entropy loss;
and (5) Sup&Cen, which trained the same network with all
the data and full labels together in a fully-supervised and
centralized manner.
Table VII reported the comparison results. With the pro-
posed regularization term and LaSA loss, our method resulted
in the best performance. Notably, compared with the Sup&Cen

Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
 GAO et al.: NEW FRAMEWORK OF SWARM LEARNING CONSOLIDATING KNOWLEDGE FROM MULTI-CENTER NON-IID DATA 2127

Fig. 7. Visualization of the segmentation results for MMWHS datasets. Arrows highlight the areas which are referred to in the text.

TABLE IX
C OMPARISON R ESULTS (D ICE S CORES ) ON P ROSTATE D ATASETS . E ACH M ETHOD WAS I MPLEMENTED W ITH 4-F OLD C ROSS VALIDATION . B OLD
T EXT D ENOTES THE B EST R ESULTS ACHIEVED BY D ISTRIBUTED L EARNING M ETHODS . ‘C ENTRALIZED ’ D ENOTES THE T RIVIAL M ETHOD
T RAINING W ITH C ENTRALIZED D ATA

TABLE X
C OMPARISON S TUDY ON F E TS D ATASETS . R ESULTS A RE E VALUATED
IN D ICE S CORE . B OLD T EXT D ENOTES THE B EST R ESULTS A CHIEVED
BY D ISTRIBUTED L EARNING M ETHODS W ITH PARTIAL L ABEL . ET:
E NHANCING T UMOR ; WT: W HOLE T UMOR ( COMPRISING ET, ED AND
NCR); TC: T UMOR C ORE ( COMPRISING ED AND ET)
method, the Dice score of our method was only 0.4% lower.
It indicates that our method could effectively handle the label
skew and feature skew challenge in the distributed learning
framework. Other methods could only slightly improve the
performance of the swarm learning method in this scenario.
This is reasonable as these methods attempt to either avoid the
divergence of training between the local and global models
by restricting the magnitude of parameter changes or forcing
their prediction to be consistent. They do not explicitly deal
with the knowledge forgetting of feature distribution and label
annotation, particularly the latter one.
2) Application to Multi-Institutional Brain Tumor Segmentation
With Label and Feature Skews: To demonstrate the generaliz-
ability to other applications, we also compared the proposed
method with SOTA methods for brain tumor segmentation on
FeTS datasets. The compared methods were similar with those
in the comparison study on M&Ms datasets.
Following the evaluation metrics of the FeTS challenge,
we reported the average Dice score on enhancing tumor (ET),
whole tumor (WT), tumor core (TC) in Table X. Compared
with other distributed learning methods, our method achieved
the best performance in average Dice score of all evaluated
regions. Moreover, our method performed comparably to the
Sup&Cen method in the segmentation of WT and TC. This
demonstrates that the proposed LaSA and FeSA loss could
effectively alleviate the issues caused by label and feature
skew in FeTS datasets. In terms of the performance on ET, the
average Dice score of our method was slightly lower than that
of Sup&Cen. The reason could be that the segmentation of ET
is difficult and there could exist non-negligible inter-observer
variability of manual annotation across centers. Thus, consol-
idating the knowledge in other centers could be misleading.
3) Application to Whole Heart Segmentation With Label Skew
Setting: To demonstrate the superiority of the proposed method
in the Non-IID scenario where only label skew exists, we com-
pared with another strategy [27](i.e., Pseudo) for partial super-
vision learning that could be applied to the swarm learning
framework. The Swarm+Pseudo method generates pseudo
labels using the global model for the supervision of unlabeled
structures.
As presented in Table VIII, the swarm learning with CE
loss and PCE loss almost failed in this scenario. This is
because these methods could not estimate an accurate loss for
unlabeled structures, and thus failed to preserve the knowledge

Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
 2128
TABLE XI
S TUDY OF THE E FFECT OF Q UANTITY S KEW FOR THE P ROPOSED N ON -IID SL M ETHOD ON M&M S D ATASET. T HE I NDEX IN THE C OLUMN OF
‘ LIMITED C ENTER ’ R EPRESENTS THE C ENTER W HICH H AS S CARCE T RAINING D ATA
of labels except 2 or 3 out of 7 structures which are provided
with annotations in a local center. Both the Swarm+Pseudo
method and our proposed Non-IID swarm learning achieved
satisfactory performance. From Table VII one can see that
the results of Swarm+Pseudo are very close to those of
Sup&Cen, which can be taken as an upper bound of federated
learning methods. Despite the limited space for improvement,
our method still obtained 1.3% higher average Dice score
than Swarm+Pseudo. The reason could be that although both
methods alleviate the forgetting of true label in local update
by distilling knowledge from the global model, the pseudo
label generated by the global model can be inaccurate. This
inaccurate label could mislead the optimization of the local
model.
The visual results are presented in Fig. 7. We can observe
that compared with the ground truth, the segmentation errors
of our method mainly occurred in Myo and PA. The reasons
are two folds. (1) Myo and PA are hard to segment, as the
results of Sup&Cen method are not accurate in them. (2)
The supervision for Myo is weak since only center A could
provide annotations for Myo. For difficult structures, more
supervision is needed. It is also observed that the performance
of a structure is not determined by whether the annotation is
available locally. For example, Center B and Center D could
not provide annotations for Myo during training. However, the
results of Myo in these two centers are visually more accurate
than that in Center A. The reason could be that our proposed
method can effectively preserve the label knowledge from
other centers during training and the performance difference
is probably due to the quality of images in each center.
4) Application to Multi-Site Prostate MRI Segmentation With
Feature Skew: To show the effectiveness of our method in
the feature skew scenario, we applied the method to seg-
mentation of the multi-site prostate T2-weighted MRI dataset,
and compared with other SOTA methods that do not apply
the personalization strategy, including (1) the standard swarm
learning method, (2) FedProx [8] and (3) FedCurv [15].
Table IX presents the results of compared global methods.
Compared with the Swarm method, the proposed Non-IID SL
achieved significant improvements in Center C and E by 4.7%
(p=0.005) and 5.8% (p=0.002), respectively. This indicates
that by applying the feature statistic alignment in the local
training, our method could learn robust feature representations
and thus improve the performance. Moreover, Non-IID SL
achieved the best performance in average Dice score among
all distributed learning methods with all p-values less than
0.05 by the paired Wilcoxon tests. The reason could be that our
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 42, NO. 7, JULY 2023
method used image distribution related regularization during
local training, while other methods applied parameter-based
regularization which could restrict the plasticity of local
models.
V. C ONCLUSION
In this paper, we have presented a new framework for
de-centerlized and privacy-protected distributed swarm learn-
ing. This framework employs the label skew-awared (LaSA)
loss, which preserves the prediction for unlabeled structures by
the global model during local training, to avoid the label skew
issue. Moreover, we have proposed the feature skew-awared
(FeSA) regularization to align feature distributions of different
centers through batch normalization statistic matching, which
alleviates the effect of feature skew. The results showed that
the proposed skew-awared functions were effective in tackling
the different distributions and could consolidate the knowledge
learned from local clients. The applications to the three tasks
of medical image segmentation have demonstrated that the
proposed method could achieve comparable results to the
fully-supervised and centralized methods.
A PPENDIX
A. Study of Quantity Skew
Here we discuss the effect of quantity skew on the proposed
method. Quantity skew, especially with highly unbalanced
numbers of subjects across centers, could degrade the perfor-
mance of federated learning methods, though this may not be
a challenge from the Non-IID problem in segmentation tasks
but different sample sizes. However, as the proposed method
adopts knowledge consolidation strategy, the effect of quantity
skew could be reduced significantly.
For demonstration, we implemented the proposed method
on M&Ms dataset with unbalanced numbers across centers.
Specifically, we reduced the numbers of training and validation
data in one of the four centers to be only 5 and 3, respectively,
and keep others unchanged. The implementation details are the
same as described in Section IV-B.
As shown in Table XI, compared with the results trained
with relatively balanced dataset, quantity skew did not lead to
obvious performance decline. The reason could be two folds.
First, our method involves knowledge consolidation strate-
gies. Centers with few training samples could also acquire
knowledge from the global model. Second, we performed data
augmentations in each center and ensured equal local training
iteration in one communication round such that the global
model will not be biased toward centers with more samples.
 GAO et al.: NEW FRAMEWORK OF SWARM LEARNING CONSOLIDATING KNOWLEDGE FROM MULTI-CENTER NON-IID DATA
R EFERENCES
[1] L. Qu, N. Balachandar, and D. L. Rubin, “An experimental study of
data heterogeneity in federated learning methods for medical imaging,”
2021, arXiv:2107.08371.
[2] N. Rieke et al., “The future of digital health with federated learning,”
NPJ Digit. Med., vol. 3, no. 1, pp. 1–7, 2020.
[3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Y. Arcas,
“Communication-efficient learning of deep networks from decentralized
data,” in Proc. Artif. Intell. Statist., 2017, pp. 1273–1282.
[4] S. Warnat-Herresthal et al., “Swarm learning for decentralized and
confidential clinical machine learning,” Nature, vol. 594, no. 7862,
pp. 265–270, 2021.
[5] S. Pati et al., “Federated learning enables big data for rare cancer
boundary detection,” 2022, arXiv:2204.10836.
[6] M. J. Sheller, G. A. Reina, B. Edwards, J. Martin, and S. Bakas, “Multi-
institutional deep learning modeling without sharing patient data: A
feasibility study on brain tumor segmentation,” in Brainlesion: Glioma,
Multiple Sclerosis, Stroke and Traumatic Brain Injuries, A. Crimi,
S. Bakas, H. Kuijf, F. Keyvan, M. Reyes, and T. van Walsum, Eds.
Cham, Switzerland: Springer, 2019, pp. 92–104.
[7] A. Guha Roy, S. Siddiqui, S. Pölsterl, N. Navab, and C. Wachinger,
“BrainTorrent: A peer-to-peer environment for decentralized federated
learning,” 2019, arXiv:1905.06731.
[8] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith,
“Federated optimization in heterogeneous networks,” in Proc. Mach.
Learn. Syst., vol. 2, 2020, pp. 429–450.
[9] P. Kairouz et al., “Advances and open problems in federated learning,”
Found. Trends Mach. Learn., vol. 14, nos. 1–2, pp. 1–210, Jun. 2021.
[10] B. Wu, S. Lyu, and B. Ghanem, “ML-MG: Multi-label learning with
missing labels using a mixed graph,” in Proc. IEEE Int. Conf. Comput.
Vis. (ICCV), Dec. 2015, pp. 4157–4165.
[11] Q. Liu, Q. Dou, L. Yu, and P. A. Heng, “MS-Net: Multi-site network for
improving prostate segmentation with heterogeneous MRI data,” IEEE
Trans. Med. Imag., vol. 39, no. 9, pp. 2713–2724, Sep. 2020.
[12] K. Gilbert et al., “Independent left ventricular morphometric atlases
show consistent relationships with cardiovascular risk factors: A UK
biobank study,” Sci. Rep., vol. 9, no. 1, pp. 1–9, 2019.
[13] C. Petitjean et al., “Right ventricle segmentation from cardiac MRI: A
collation study,” Med. Image Anal., vol. 19, no. 1, pp. 187–202, 2015.
[14] N. Shoham et al., “Overcoming forgetting in federated learning on non-
IID data,” 2019, arXiv:1910.07796.
[15] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “FedBN: Federated
learning on non-IID features via local batch normalization,” 2021,
arXiv:2102.07623.
[16] K. Bonawitz et al., “Practical secure aggregation for privacy-preserving
machine learning,” in Proc. ACM SIGSAC Conf. Comput. Commun.
Secur., Oct. 2017, pp. 1175–1191.
[17] X. Li, Y. Gu, N. Dvornek, L. H. Staib, P. Ventola, and J. S. Duncan,
“Multi-site fMRI analysis using privacy-preserving federated learning
and domain adaptation: ABIDE results,” Med. Image Anal., vol. 65,
Oct. 2020, Art. no. 101765.
[18] A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated
learning with theoretical guarantees: A model-agnostic meta-learning
approach,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020,
pp. 3557–3568.
[19] T. Shen et al., “Federated mutual learning,” 2020, arXiv:2006.16765.
[20] T. Li, S. Hu, A. Beirami, and V. Smith, “Ditto: Fair and robust federated
learning through personalization,” in Proc. Int. Conf. Mach. Learn.,
2021, pp. 6357–6368.
[21] L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploit-
ing shared representations for personalized federated learning,” 2021,
arXiv:2102.07078.
[22] K. James et al., “Overcoming catastrophic forgetting in neural networks,”
Proc. Nat. Acad. Sci. USA, vol. 114, no. 13, pp. 3521–3526, Mar. 2017.
[23] M. Delange et al., “A continual learning survey: Defying forgetting in
classification tasks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44,
no. 7, pp. 3366–3385, Jul. 2021.
Authorized licensed use limited to: Chaitanya Bharathi Institute of Tech - HYDERABAD. Downloaded on September 09,2023 at 06:07:55 UTC from IEEE Xplore. Restrictions apply.
2129
[24] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic
intelligence,” in Proc. Int. Conf. Mach. Learn., 2017, pp. 3987–3995.
[25] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars,
“Memory aware synapses: Learning what (not) to forget,” in Proc. Eur.
Conf. Comput. Vis. (ECCV), 2018, pp. 139–154.
[26] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, “PodNet:
Pooled outputs distillation for small-tasks incremental learning,” in Proc.
Eur. Conf. Comput. Vis., 2020, pp. 86–102.
[27] A. Douillard, Y. Chen, A. Dapogny, and M. Cord, “PLOP: Learn-
ing without forgetting for continual semantic segmentation,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021,
pp. 4040–4050.
[28] Y. Lu, X. Huang, Y. Dai, S. Maharjan, and Y. Zhang, “Blockchain
and federated learning for privacy-preserved data sharing in industrial
IoT,” IEEE Trans. Ind. Informat., vol. 16, no. 6, pp. 4177–4186,
Jun. 2020.
[29] I. Szita and A. Lörincz, “Learning tetris using the noisy cross-
entropy method,” Neural Comput., vol. 18, no. 12, pp. 2936–2941,
Dec. 2006.
[30] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for
dense object detection,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV),
Oct. 2017, pp. 2980–2988.
[31] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-Net: Fully convolutional
neural networks for volumetric medical image segmentation,” in Proc.
4th Int. Conf. 3D Vis. (DV), Oct. 2016, pp. 565–571.
[32] G. Shi, L. Xiao, Y. Chen, and S. K. Zhou, “Marginal loss and exclusion
loss for partially supervised multi-organ segmentation,” Med. Image
Anal., vol. 70, May 2021, Art. no. 101979.
[33] F. Cermelli, M. Mancini, S. Rota Bulo, E. Ricci, and B. Caputo,
“Modeling the background for incremental learning in semantic seg-
mentation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2020, pp. 9233–9242.
[34] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in Proc. Int. Conf.
Mach. Learn., 2015, pp. 448–456.
[35] Y. Li, N. Wang, J. Shi, X. Hou, and J. Liu, “Adaptive batch nor-
malization for practical domain adaptation,” Pattern Recognit., vol. 80,
pp. 109–117, Aug. 2018.
[36] V. M. Campello et al., “Multi-centre, multi-vendor and multi-disease
cardiac segmentation: The M&Ms challenge,” IEEE Trans. Med. Imag.,
vol. 40, no. 12, pp. 3543–3554, Dec. 2021.
[37] S. Pati et al., “The federated tumor segmentation (FeTS) challenge,”
2021, arXiv:2105.05874.
[38] X. Zhuang et al., “Evaluation of algorithms for multi-modality whole
heart segmentation: An open-access grand challenge,” Med. Image Anal.,
vol. 58, Dec. 2019, Art. no. 101537.
[39] U. Baid et al., “The RSNA-ASNR-MICCAI BraTS 2021 benchmark
on brain tumor segmentation and radiogenomic classification,” 2021,
arXiv:2107.02314.
[40] G. A. Reina et al., “OpenFL: An open-source framework for federated
learning,” 2021, arXiv:2105.06413.
[41] N. Bloch et al., “NCI-ISBI 2013 challenge: Automated segmentation of
prostate structures,” Cancer Imag. Arch., vol. 370, p. 6, Oct. 2015.
[42] G. Lemaître, R. Martí, J. Freixenet, J. C. Vilanova, P. M. Walker, and
F. Meriaudeau, “Computer-aided detection and diagnosis for prostate
cancer based on mono and multi-parametric MRI: A review,” Comput.
Biol. Med., vol. 60, pp. 8–31, May 2015.
[43] G. Litjens et al., “Evaluation of prostate segmentation algorithms for
MRI: The PROMISE12 challenge,” Med. Image Anal., vol. 18, no. 2,
pp. 359–373, 2014.
[44] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolu-
tional networks for biomedical image segmentation,” in Proc.
Int. Conf. Med. Image Comput. Comput.-Assist. Intervent, 2015,
pp. 234–241.
[45] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein,
“NnU-Net: A self-configuring method for deep learning-based biomed-
ical image segmentation,” Nature Methods, vol. 18, no. 2, pp. 203–211,
Dec. 2020.