See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/374350920

BM-Seg: A new bone metastases segmentation dataset and ensemble of CNN-

based segmentation approach

Article in Expert Systems with Applications · October 2023

DOI: 10.1016/j.eswa.2023.120376

CITATIONS READS
0 25

7 authors, including:

Marwa Afnouch Olfa Gaddour

Ecole Nationale d'Ingénieurs de Sfax University of Sfax
2 PUBLICATIONS 0 CITATIONS 15 PUBLICATIONS 922 CITATIONS

SEE PROFILE SEE PROFILE

Fares Bougourzi Ihsen Alouani

Università del Salento Queen's University Belfast
25 PUBLICATIONS 301 CITATIONS 87 PUBLICATIONS 813 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Fares Bougourzi on 04 November 2023.

The user has requested enhancement of the downloaded file.

BM-Seg: A New Bone Metastases Segmentation Dataset
and Ensemble of CNN-Based Segmentation Approach
Marwa Afnouch∗a,b , Olfa Gaddourb , Yosr Hentatic , Fares Bougourzie ,
Mohamed Abidb , Ihsen Alouania,d , Abdelmalik Taleb Ahmeda
a
Laboratory of IEMN, CNRS, Centrale Lille, UMR 8520 , Univ. Polytechnique
Hauts-de-France, F-59313, Valenciennes, France
b
CES-laboratory, National Engineering School of Sfax, University of
Sfax, 3038, Sfax, Tunisia
c
Department of Radiology, Hedi Chaker University Hospital, 3029, Sfax, Tunisia
d
CSIT, Queen’s University Belfast, BT3 9DT, Belfast, United Kingdom
e
National Research Council of Italy, Institute of Applied Sciences and Intelligent
Systems, 73100 , Lecce, Italy

Abstract
In recent years, Machine Learning (ML) approaches have shown promising
results in achieving many medical imaging analysis tasks. In particular, Bone
Metastases (BM) analysis is getting significant interest from both medical
and computer vision communities due to its critical and challenging aspect.
Despite the research effort, detecting BM is still an open problem primarily
because of the lack of available datasets. This is due to two main obstacles:
(i) the huge time required for data collection and annotation and (ii) data
privacy constraints. To address these challenges, we propose BM-Seg, a new
dataset for BM segmentation from CT-scans. Our BM-Seg dataset consists
of 1517 CT images from 23 patients, where BM and bone regions are labeled
by three expert radiologists. BM-Seg is constructed to cover the diversity of
bone metastases in terms of location, organ, and severity.
Moreover, we propose a new CNN-based approach for BM segmentation
in which two main contributions are presented. First, we introduce a Dual-
Decoder Attention Unet++ (DAUnet++) architecture based on Unet++

∗
Corresponding author
E-mail addresses: marwa.afnouch@enis.tn, olfa.gaddour@enis.tn, yosr.hentati@yahoo.fr,
faresbougourzi@gmail.com, med.abid@enis.tn, Abdelmalik.Taleb-Ahmed@uphf.fr, ih-
sen.alouani@uphf.fr

Preprint submitted to Expert Systems with Applications March 8, 2023

and attention gates with double decoders to segment BM and bone regions,
simultaneously. Second, we use an Ensemble of trained DAUnet++ models
(EDAUnet++) to optimize the segmentation performance. Our experiments
show that EDAUnet++ architecture achieves higher performance compared
to the state-of-the-art approaches for different evaluation metrics. In addi-
tion, we proceed to an ablation study, which shows the effectiveness of each
of the proposed approach’s components.
We hope that the proposed dataset and approaches will help the commu-
nity progress toward more efficient automatic BM segmentation systems.
Keywords: Bone Metastases, CT-scans, Semantic Segmentation, Ensemble,
Dataset, Machine Learning, Convolutional Neural Network

1. Introduction
Metastases refer to a group of abnormal cells that develop outside of
their normal borders and spread to other organs. In particular, BM is a
cancer that originates in the bodys organs such as breast, lung, or prostate,
and spread to the bone. In 90 % of cancer deaths, metastases were found
to be a contributing factor Steeg (2006); Seyfried and Huysentruyt (2013).
Bone ranks third behind liver and lung, among the most common sites of
metastasis Macedo et al. (2017). Statistics show that more than 70 % of
patients with breast and prostate cancer have BM O’Sullivan (2015). In this
regard, the early detection of bone cancers is essential to make the right
decision Coleman et al. (2020).
Tracking BM requires medical imaging to diagnose diseases and determine
treatment effectiveness. Several medical imaging modalities have been used,
including Computed tomography (CT), MRI (Magnetic Resonance Imaging),
bone scans, and PET scans, where each imaging modality has its advantages
and drawbacks Turpin et al. (2020). In particular, CT-scans have a high
spatial resolution, which allows the detection of large bone lesions as well
as mild and small ones.O’Sullivan (2015). They also offer a simultaneous
evaluation of primary bone and BM lesions. Besides, CT-scans are commonly
available at a relatively low-cost Hammon et al. (2013). In the clinical setting,
CT-scans are the most commonly used imaging modality for both baseline
staging tests and serial surveillance of patients with cancer Heindel et al.
(2014).

2
 However, the detection of BM from CT-scans is both challenging and
time-consuming for the following reasons: (i) since bones are throughout
the body, radiologists must examine all slices Noguchi et al. (2020); (ii)
the radiological appearance of BM depends on lesion types Heindel et al.
(2014). Therefore, no single window parameter can adequately represent all
bone metastases; and (iii) benign mimics such as fractures, bone islands, and
degenerative changes can be confused with BM lesions, which complicate the
diagnosis Vandemark et al. (1992). Computer-Aided Diagnosis (CAD) can
assist radiologists in finding small lesions that might otherwise be missed
Jadon (2020). Consequently, there is a high demand for CAD systems for
BM on CT-scans.
Recently, Deep Learning methods have gained much attention to solve
several visual-related tasks Bougourzi et al. (2020); Litjens et al. (2017);
Chakraborty and Mali (2021); Bougourzi et al. (2022). In particular, Con-
volutional Neural Networks (CNNs) achieved state-of-the-art performance
on various types of image recognition problems Vantaggiato et al. (2021);
Bougourzi et al. (2021). Among these problems, Semantic Segmentation is
one of the most studied topics in the last years, where each pixel in an im-
age is assigned to one of a set of labels Hiramatsu et al. (2018). In the
medical imaging field, automatic segmenting of infection lesions is a critical
step in diagnosing and understanding the disease Diniz et al. (2022); Allah
et al. (2023). In recent years, MRI and CT imaging modalities have been
widely used alongside machine learning approaches for automatic lesion seg-
mentation da Cruz et al. (2022); Lei et al. (2021). However, the automated
segmentation of BM using CT-scans has received relatively little attention
Noguchi et al. (2022); Chmelik et al. (2018).
To the best of our knowledge, there is no publicly available dataset associ-
ated with BM segmentation. This work aims to create a benchmark dataset
for BM segmentation from CT-scans. Moreover, we propose a new CNN-
based approach for BM segmentation to further improve the segmentation
performance compared with state-of-the-art architectures.
The main contributions of this work can be listed as follows:
1. We introduce BM-Seg dataset for segmenting BM from CT-scans. The
BM-Seg dataset is publicly available at https://BMseg shortlink.edu
2. We present DAUnet++, a new architecture derived from Unet++ that
combines attention gates, and dual decoders. DAUnet++ efficiently
segments both BM and bone regions.

3
 3. To boost the BM segmentation, we propose an ensemble approach
which we baptize EDAUnet++. In this approach, five trained DAUnet++
models are used to enhance predictions of the single models, by aver-
aging the predicted probabilities from different estimators.

4. We conduct extensive experiments with the BM-Seg dataset, with fa-

vorable results compared to other baseline CNN-based segmentation
architectures, and we run an ablation study for a comprehensive anal-
ysis of our approach. The codes of the proposed approach will be
released at: https://github.com/faresbougourzi/EDAUnetplus.

This paper is organized as follows. Section 2 briefly reviews recent state-of-

the-art works for BM analysis. In section 3, the proposed BM-Seg dataset
is illustrated. Section 4 presents our proposed approach. The experimental
setups and results are displayed and analyzed in sections 5 and 6, respectively.
Finally, section 7 concludes the paper and discusses future work.

2. Related Work
In recent years, there has been considerable interest in the automatic as-
sessment of BM from different medical imaging modalities such as bone scans,
CT-scans, and SPECT. The state-of-the-art work focuses on two main tasks:
(i) classifying BM scans into normal and abnormal cases and (ii) segmenting
lesions. Recently, several recent research works have dealt with the classifi-
cation of BM by using deep learning methods Papandrianos et al. (2020b);
Aslantas et al. (2016); Masoudi et al. (2021); Pi et al. (2020); Li et al. (2020);
Guo et al. (2022); Apiparakoon et al. (2020). On the other hand, automated
segmentation of metastatic lesions is still in its infancy. Shimizu et al. (2020)
proposed an image interpretation system for the skeleton segmentation and
extraction of hotspots of a metastatic bone from a whole-body bone scinti-
gram based on deep learning. Zhang et al. (2021) developed a segmentation
algorithm based on UNet with an attention mechanism for SPECT bone
segmentation, which can automatically identify the location of BM. Moreau
et al. (2020) compared different approaches of bones and bone metastatic
lesions in breast cancer segmentation. Two deep learning methods based on
U-Net were developed and trained to segment either both bones and bone
lesions or bone lesions alone on PET/CT images. Later, Fan et al. (2021)
proposed a deep learning algorithm for spinal metastasis from lung cancer.

4
 Table 1: Summary of existing Bone Metastasis Datasets

Literature Data Modality Dataset description Task Availability

586 scans[368 BM;18 Normal]
Papandrianos et al. (2020b) Bone scan Classification Private
507 patients
778 scans [328 BM; 271 degenerative; 179 normal]
Papandrianos et al. (2020c) Bone scan Classification Private
817 patients
Papandrianos et al. (2020a) Bone scan 408 scans[ 221 BM; 187 Normal; 382] patient Classification Private
15,474 images
Pi et al. (2020) Bone scan Classification Private
13.811 patients [9595 benign; 5879 malignant]
1088 labled chest ; 18.560 unlabeled Segmentation
Apiparakoon et al. (2020) Bone scan Private
9.280 patients Classification
Shimizu et al. (2020) Bone scan 246 bone scans segmentation Private
Cheng et al. (2021) Bone scan 205 scans[ 100 BM; 105 Normal] Classification Private

5
Aslantas et al. (2016) Bone scan 130 images [ 30 Normal;100 BM] 60 patients Classification Private
9133 images[ 2991 BM; 6142 Normal]
Han et al. (2021) Bone scan Classification Private
5342 patients
2.880 scans
Masoudi et al. (2021) CT Classification Private
114 patients [41 BM]
269 BM scans
Noguchi et al. (2022) CT Segmentation Private
169 patients
Moreau et al. (2020) PET/CT 24 patients Segmentation Private
260 scans
Cao et al. (2023) SPECT Segmentation Private
130 patients
76 patients
Lin et al. (2020) SPECT Classification Private
152 scans
23 patients
The proposed dataset CT Segmentation Public
1517 images
 They proposed a dilated convolutional U-Net (DC-U-Net) model to seg-
ment the energy/spectral CT images. More recently, Noguchi et al. (2022)
used CT images and proposed a segmentation approach based on three convo-
lutional neural networks (CNNs): a 2D UNet-based network for segmenting
bones, a 3D UNet-based network for segmenting candidate regions, and a
3D ResNet-based network for reducing false-positive results. Although the
state-of-the-art methods achieved promising results, their impact is limited
by using private datasets, which prevents the community from exploiting the
data and building upon their findings. In addition, existing works have used
several imaging modalities including MRI, bone scan, and PET while few
works used CT-scan despite its distinct advantages. Table 1 shows a com-
parison of existing BM datasets. Based on Table 1, the main limitation of
existing datasets is their small size, which can lead to overfitting. To over-
come these problems, we created a new BM segmentation dataset and made
it publicly available to the research community.

3. BM-Seg Dataset
In this section, the following steps to create the BM-Seg dataset are de-
scribed in Figure 1.

Figure 1: Process of building BM-Seg Dataset

6
 Initially, a contrast substance is injected into the patient’s veins to high-
light any bone lesions. Once a patient exit the CT scanner, the CT images
are sent to three radiologists for examination. After interpreting data by
doctors, the regions of interest (RoIs) that represent the bone and lesion re-
gions for each CT-scan slice are marked using the Apeer tool Zeiss (2021).
At the end of the process of building BM-Seg dataset, we obtain both bone
and lesion masks for each BM image.

3.1. Data Collection and Labelling

Our BM-Seg dataset consists of CT-scans from 23 metastatic bone pa-
tients with 9 female and 14 male subjects, ranging from 18 to 83 years old.
All CT-scans were collected retrospectively from UHC (University Hospi-
tal Center) Hedi Chaker, Sfax, Tunisia. The data had been collected from
November 2020 to June 2022. Each CT-scan is reviewed by three expert ra-
diologists using the Radiant Dicom Viewer tool Medixant (2009). Based on
the location of bone pain, the patients health, and any prior trauma history,
two doctors annotated the slices of each CT-scan as infected or not infected.
Then, an expert radiologist (more than 20 years of diagnostic imaging expe-
rience) supervised the annotated slices that had been originally evaluated by
the two other doctors. Finally, the radiologists selected the regions that cor-
respond to BM lesions in each CT-scan infected slice. The examination was
excluded if there was no clear diagnosis agreement between the three radiol-
ogists. Figure 2 shows examples of BM-Seg images with their corresponding
annotated lesions.

Figure 2: BM examples from BM-Seg. Eight slices from different patients are reported.
The first row contains BM images, and the second row contains an annotation of each
example (colors refer to lesion locations).

7
 The selected slices were converted to JPEG format, then bone and BM
regions were manually segmented to extract ground truth (GT) masks using
Apeer software. Figure 3 shows four extracted CT-scan images from BM-seg
with their corresponding ground truth masks. The selected images in the
figure belong to different sites in the human skeleton and have various BM
lesion sizes. As the labelling process is time consuming, 70 infected slices
were randomly selected from each CT-scans. On the other hand, in the CT-
scans that have less than 70 infected slices, all infected slices are selected. In
total, 1517 slices were annotated by creating the bone metastasis and bone
masks.

Figure 3: Examples of CT images with various BM sizes and different positions in the
bone skeleton.

Table 2 reports the types of primary cancers and incidences of lesions in

our dataset. Although the majority of patients have lung tumors as their
primary cancers, the vertebrae represent the most commonly affected bone
area by lung tumors, so the vertebrae slices are the most prevalent. Moreover,
BM-Seg followed a real-world distribution without excluding any cases, in

8
contrast to prior studies that excluded misleading examples and concentrated
on only one type of primary cancer. Thus, a trained system that uses BM-Seg
is expected to be more appropriate for routine clinical applications.

Table 2: Types and incidences of primary tumors among 1517 BM examination

Type Lung Breast Kidney Prostate Stomach Liver Others

Primary Tumors
Nb 8 6 1 2 1 1 4
Type Full Bone Vertebrae Pelvis Rib Humerus Foot
Lesion location
Nb 490 639 128 112 175 78

3.2. BM-Seg Challenges

To evaluate bone metastasis from CT-scan, an expert radiologist needs to
look at both clinical and anatomical context. The most accurate assessment
comes from comparing and contrasting CT-scan with other medical imaging
modalities such as MRI and SPECT scans. The main challenges that face
the radiologist when examining the CT-scans are:

• Bone lesions may appear not just in one region but also along several
sites such as ribs, pelvis, and spine. To identify and distinguish bone
lesions, human professionals should scrutinize all the CT-scan slices.

• BM are classified as osteoblastic, osteolytic, and mixed lesions Gurkan

et al. (2019). While osteolytic lesions are responsible for bone resorp-
tion and show a lower density than the normal cancellous bone, os-
teoblastic lesions cause too many bone cells to form, which makes the
bone very dense. According to Hammon et al. (2013), it is challenging
and time-consuming to detect bone lesions at an early stage on CT im-
ages especially when a variety of benign bone lesions with an osteolytic
appearance are present. As a result, each lesion type has a different
appearance, which complicates the annotation process and makes the
segmentation task more challenging.

• Benign processes such as inflammation, degenerative activity, fracture,

and injury will show hotspots. In addition, primary malignant and
benign bone tumors will show similar appearances, i.e increase in ra-
diotracer uptake.

9
Figure 4: Osteolytic lesion appearance (circles in green) vs Disc appearance (circles in
red) in BM-Seg dataset

Figure 5: Osteoblastic lesion appearance (circle in green) vs Degenerative bone appearance

(circle in red) in BM-Seg dataset.

To overcome the above challenges, annotation by different radiologists is

essential to obtain redundant information or complementary data that can
compensate for the challenges present in each CT-scan. Figures 4 and 5 high-
light the most introduced challenges in our dataset. As shown in Figure4,

10
it is difficult to distinguish between disc in CT-scan and bone lesion, espe-
cially osteolytic lesions where radiologists have to spend considerable time
verifiying one patient scan. Figure5 shows an additional issue in BM seg-
mentation which is the difficulty to differentiate between osteoblastic lesions
and degenerative bones as they have similar appearances.

4. Proposed Approach
In this section, we will describe the design of our proposed EDAUnet++
approach for BM segmentation.

4.1. Attention Unet++ (AttUnet++)

As described in Li et al. (2020), AttUnet++ consists of an encoder, a
decoder, and nested layers. Similarly to Unet architecture Ronneberger et al.
(2015), AttUnet++ has a symmetrical encoder and decoder on each side. The
key difference between AttUnet++ and Unet is that AttUnet++ replaces
the skip connections with nested layers. Each nested layer extracts semantic
information through multiple attention gates and convolution layers. Each
convolution layer in nested layers as in the encoder and the decoder is unified
and is designated as Basic Convolutional Block (BCBlock). The BCBlock in
Li et al. (2020) consists of two 3 × 3 convolutional blocks and each is followed
by the ReLU activation function. The encoder of AttUnet++ consists of
consecutive BCBlocks and maxpooling layers. As in the AttUnet++ decoder,
each BCBlock is followed by a linear up-sampling layer. On the other hand,
the nested layers consist of BCBlocks, up-sampling, attention gates, and skip
connections.

4.2. DAUnet++ Architecture

We design DAUnet++ model based on AttUnet++ architecture to seg-
ment both BM and bone regions simultaneously. As shown in Figure 6,
DAUnet++ architecture contains an encoder and two decoders. The en-
coder is composed of BCBlocks, where each BCBlock is followed by 2 × 2
max-pooling layer, expect the last BCBlock. For the convolutional layers in
the encoder blocks, we use 32, 64, 128, 256, and 512 filters, doubling the num-
ber of feature maps after each block. This structure allows the encoder to be
able to learn higher-level features. These features are exploited by the first
decoder to segment the bone regions, while the second decoder exploits them
to segment BM regions. In each decoder, the number of channels is reduced

11
from 512 to 256, 128, 64, 32, and 1, respectively, using five BCBlocks. To
transmit the context information that the encoder has recovered to the de-
coders of the relevant layers, we used nested layers that allow the extraction
of more efficient hierarchical features.

Figure 6: DAUnet++ Architecture

Similar to AttUnet++, the nested layers in DAUNet++ consist of three

components: BCBlocks, attention gates, and two types of links.

• The BCBlock is applied to the output of the previous layer to trans-

form the shallower, lower, and coarse-grained features into the deeper,
semantic, and fine-grained features.

• Attention gates are integrated between BCBlocks to find the most cru-
cial information for the segmentation task before propagating it to the
decoder part. Each attention gate combines the output of the previous
BCBlock with the output of the corresponding up-sampled low-density
block.

12
 • Up-sampling and skip connections are used to extract representative
features and eliminate semantic gaps between features. The outputs
of the attention gates are concatenated with the output from the up-
sample of the lower hop path to reach the next nested layer.
Let xi,j represent the output of the BCBlock X i,j , where i refers to the
feature depth in the encoder and j to the depth of the convolution layer in
the nested block. We define the extracted feature map of the convolution
layer xi,j as follows:
(
H[xi−1,j ] j=0
xi,j = R j−1 i,n i+1,j−1
(1)
H[ n=0 Ag(x ), U p(x )] j > 0

where H[.] is a cascaded operation that includes convolution, BN, ReLU

activation function, and residual skip connection. Ag () denotes the attention
gate function, and Up() is a linear up-sampling.

Figure 7: The used BCBlock structure of the proposed DAUnet++

In our DAUnet++ architecture, each BCBlock is modified by adding

Batch Normalization (BN) and residual skip connection. In Figure 7, we
illustrate the BCBlock structure. The residual skip connection is a 1 by 1
convolution layer followed by ReLU activation function and BN to convert
the input feature map to the desired output feature map. Experimentally,
the modified BCBlock significantly improved performance over the BCBlock
presented in AttUnet++ Li et al. (2020) as detailed later in Section 5.

4.3. EDAUnet++
Ensemble models are created in ML by merging the predictions of vari-
ous independent models to enhance the overall predictions. The aim of the
Ensemble method is to reduce generalization errors in ML algorithms and

13
 Figure 8: The proposed ensemble framework for BM segmentation

to improve the segmentation outcome by taking lessons from the results of

individual models. To improve the performance of our proposed DAUnet++
architecture, we propose an ensemble method composed of five DAUnet++
models trained from scratch as illustrated in Figure 8. The trained mod-
els are used to predict individual predictions then ensemble is performed by
averaging the five models predictions.

4.4. Hybrid Loss Function

To enforce DAUnet++ to pay more attention to lesion segmentation than
bone segmentation, we design a hybrid loss function combining bone loss
and lesion loss based on Binary Cross Entropy (BCE) loss function Jadon
(2020). BCE, which is known to be suitable for classification and semantic
segmentation, is expressed as:

LBCE (y, ŷ) = −(ylog(ŷ) + (1 − y)log(1 − ŷ)) (2)

where y ∈ [0, 1] is the prediction, and ŷ ∈ [0, 1] is the ground truth.

Let α1 = 0.7 and α2 = 0.3 are respectively the weights of lesion loss
LossL () and bone loss LossB (), the proposed hybrid loss function is defined
in equation 3.
Loss = α1 · LossL + α2 · LossB (3)

14
 LossL = LBCE (yL , yˆL ) (4)
LossB = LBCE (yB , yˆB ) (5)
where yL ,yB are the bone and metastasis predictions, respectively. yˆL is the
lesion mask and yˆB is the bone mask.

5. Experiments and Results

5.1. Methodology
5.1.1. Experimental Setups
The model is implemented based on the PyTorch framework and is trained
on a piece of Nvidia Quadro GTX 5000 GPU. A total of 60 epochs are trained
for the model with a learning rate of 0.01 for the first 20 epochs, 0.005 for the
second 20 epochs and finally 0.001 for the last 20 epochs. A batch size of 8 is
used in the training phase. As a loss function, the hybrid loss is used for BM
and bone regions segmentation while BCE loss is used for the comparison
methods.

5.1.2. Evaluation Metrics

Various evaluation metrics are used to assess the performance of the pro-
posed BM segmentation model on BM-Seg. The evaluation metrics include
F1-score (equation 6), Accuracy (equation 7), Precision(Prec) (equation 8),
Specificity (Spec) (equation 9), Sensitivity (Sens) (equation 10), Intersection-
Over-Union (IoU)(equation 11) and Dice Coefficient (Dice)(equation 12).
As for bone metastasis detection, a TP (True Positive) indicates that
the image label is malignant and is correctly classified. FP (False Positive)
means benign label but malignant classification. TN (True Negative) implies
that the image is benign and classified as such. FN (False Negative) signifies
that the image is classified as benign, despite the label being malignant.
|A ∩ B| 2 × TP
F1-score = 2 × = (6)
(|A| + |B|) (2 × T P + F P + F N )

TP + TN
Accuracy = (7)
(T P + T N + F P + F N )

TP
P recision = (8)
(T P + F P )

15
 TN
Specif icity = (9)
(T N + F P )

TP
Sensitivity = (10)
(T P + F N )

(T arget ∩ P redicted)
IoU = (11)
(T arget ∪ P redicted)

N
1 X T Pi
Dice-score = 2× (12)
N i=1 (2 × T Pi + F Pi + F Ni )

Where N is the number of testing images.

5.1.3. Comparison with the State-of-the-art Approaches

The experiments included two sets of model comparisons to examine
model performance from various angles. First, we compare EDAUnet++
with Unet Ronneberger et al. (2015) and its variants including AttUnet Ok-
tay et al. (2018), Unet++ Zhou et al. (2018) and AttUnet++Li et al. (2020).
Furthermore, we compare the performance of our approach with the sequen-
tial scenario Noguchi et al. (2022), in which bone regions are segmented first
then BM are segmented from the bone regions. To this end, three sequential
combinations are tested, which are denoted by sequential(Unet, AttUnet), se-
quential(Unet, Unet++) and sequential(Unet, AttUnet++), where the first
architecture is for bone segmentation and the second one is for bone metas-
tasis segmentation.

5.2. Experimental results

Table 3 lists the quantitative experimental performance of EDAUnet++
compared to the state-of-the-art models on BM-Seg. Our proposed approach
obtains the best performance in terms of F1-score, Dice, and IoU, which are
the most commonly used metrics for segmentation tasks. According to our
findings, F1-score was improved by 4.21%, 4.26%, 3.92% and 3.38% compared
with Unet, AttUnet, Unet++, and AttUnet++, respectively. As for Dice
score, we obtained gains of 4.7%, 5.28%, 5.06% and 4.68% against Unet,

16
AttUnet, Unet++ and AttUnet++, respectively. EDAUnet++ outperforms
Unet, AttUnet, Unet++, and AttUnet++ by 5.9%, 6.06%, 5.6%, 4.86%
for IoU metric. Concerning accuracy, specificity, sensitivity, and precision
metrics, EDAUnet++ still provides the highest results.

Table 3: Comparative results between state-of-the-art models and EDAUnet++

Model F1-score Dice IoU Acc Spec Sens Prec

Unet [Ronneberger et al. (2015)] 79.46 72.26 65.93 98.91 78.75 99.46 80.22
AttUnet [Oktay et al. (2018)] 79.41 71.76 65.86 98.90 79.20 99.44 79.64
Unet++ [Zhou et al. (2018)] 79.74 71.99 66.31 98.92 79.41 99.46 80.16
AttUnet++ [Li et al. (2020)] 80.28 72.36 67.06 98.95 79.66 99.48 80.94
DAUnet++ 82.27 75.70 69.89 99.05 81.88 99.53 82.68
EDAUnet++ 83.67 77.05 71.92 99.13 99.59 82.68 84.68

Table 4 shows a comparison between EDAUnet++ and the three se-

quential combinations. It should be noted that EDAUnet++ performs bet-
ter than sequential(Unet,AttUnet) in terms of F1-score by 6.25%, Dice by
10.16%, and IoU by 8.76%. Moreover, our approach outperforms sequential
(Unet,Unet++) by 6.13%, 9.29%, and 8.6% for F1-score, Dice, and IoU re-
spectively. An improvement of 5.94%, 8.44%, and 8.34% for F1-score, Dice,
and IoU relative to EDAUnet++ was observed over sequential (Unet, At-
tUnet++). Therefore, we can emphasize that EDAUnet++ achieved higher
performance compared to the selected sequential architectures.

Table 4: Comparative results between combined models and EDAUnet++ on BM-Seg.

Model F1-score Dice IOU Acc Spec Sens Prec

Sequential (Unet, AttUnet) 77.41 66.88 63.16 98.83 74.54 99.50 80.63
Sequential (Unet, Unet++) 77.54 67.75 63.32 98.83 75.14 99.49 80.12
Sequential (Unet, AttUnet++) 77.73 68.60 63.58 98.83 76.33 99.45 79.24
EDAUnet++ 83.67 77.05 71.92 99.13 99.59 82.68 84.68

To compare the performance of our approach and the state-of-the-art

approaches on the level of each fold performance, Table 5 summarizes the
obtained results of F1-score. These experiments show that EDAUnet++ not
only encompasses a higher mean F1-score than the baseline models but also
conjointly has a higher F1-score and a lot of reliability for every fold.

17
 Table 5: F1-score of 5-fold cross-validation experiments for BM segmentation

Model Fold1 Fold2 Fold3 Fold4 Fold5 Mean

Unet 80.41 77.52 79.27 80.65 79.44 79,46
AttUnet 79.17 77.79 79.04 80.63 80.39 79,41
Unet++ 80.12 80.21 78.44 80.27 79.67 79.74
AttUnet++ 80.45 80.35 79.61 80.31 80.67 80.28
Sequential( Unet, AttUnet) 78.23 76.81 75.86 78.67 77.49 77.41
Sequential( Unet, Unet++) 76.49 77.58 78.93 76.84 77.84 77.54
Sequential( Unet, AttUnet++) 78.19 78.26 76.04 77.87 78.29 77.73
EDAUnet++ 84.19 83.49 83 83.96 83.69 83.67

6. Discussion and Ablation Study

BM detection on CT-scan is a prevalent yet challenging task for radiolo-
gists. Moreover, the diagnosis of BM by imaging is further complicated by
the rarity of BM available datasets. In this study, we created our BM-Seg
dataset and we used a deep learning algorithm combining double decoders
and ensemble method to overcome these issues. As mentioned in Table 4, the
F1-score, Dice and IoU of segmenting BM were 83.67%, 77.05% and 71.92%,
respectively, which were higher than the results of baseline models. The vi-
sual comparison of EDAUnet++ with state-of-the-art algorithms is shown in
Figure 9. The images from left to right show the raw afflicted CT-scan im-
ages and the segmentation results of Unet, AttUnet, Unet++, AttUnet++,
the proposed EDAUnet++, and the GT mask. Masks comparison clearly
show that EDAUnet++ has fewer false detection pixels compared to base-
line algorithms, making it more successful at BM segmentation. Moreover,
EDAUnet++ effectively suppresses background noise and retains lesion de-
tails. In Figure 9, row 1 shows that EDAUnet++ is less affected by noise in
the bone region as it can remove noise in irrelevant locations. In addition,
rows 2 and 3 illustrate how Unet, AttUnet, Unet++, and AttUnet++ meth-
ods failed to detect small-sized lesions on complex BM-Seg images. Although
small-sized lesions details can be easily lost, EDAUnet++ exhibited better
preservation of these details. Similarly, compared to the reference models,
EDAUnet++ shows a lower misjudgment rate in images where the bone is
totally infected as can be seen in rows 4 and 5.

18
 Figure 9: Visual comparison of BM segmentation results

In Figure 10, the output results of five runs of DAUnet++ are compared
with the output of the ensemble method. Clearly, EDAUnet++ consistently
provides the best result when it comes to F1-score, regardless of the location
and size of the lesion. As an example, in the second row, EDAUnet++
reached 77.44% of F1-score while DAUnet++ did not exceed 75% in the
five runs (M1-M5). Particularly for small-scale metastases as shown in the
third row, our model is able to extract more lesion features with greater
F1-score, demonstrating its ability to learn less redundant information and
integrate global context. Consequently, metastatic bone segmentation can
be greatly improved with EDAUnet++ by extracting more comprehensive
background features, providing more detailed information, and obtaining a
more comprehensive semantic representation.

19
 Table 6: Ablation results on BM-Seg dataset

Ablation Dataset
Architecture Att DD En F1-score Dice IoU
Unet++ ✗ ✗ ✗ 79.74 71.99 66.31
AttUnet++ ✓ ✗ ✗ 80.27 72.63 67.06
DAUnet++ ✓ ✓ ✗ 82.27 75.70 69.89
EDAUnet++ ✓ ✓ ✓ 83.67 77.05 71.92

Figure 10: Comparison of segmentation performance obtained by five DAUnet++ models

(M1-M5) and EDAUnet++ model on BM-Seg (F1=F1-score, GT=Ground Truth).

For a comprehensive analysis, we conduct ablation experiments to eval-

uate the efficiency of the ensemble method and the dual-decoder structure
proposed in EDAUnet++ model. An improved AttUNet++ model is em-
ployed as the baseline with an attention mechanism which allow the decoder
in the Unet++ architecture to use the most relevant parts from the encoder.
Additionally, the dual decoder captures both substantial and minute visual
structures, which is useful for segmenting different sizes of lesions. The ab-
lation results are listed in Table 6, from which we can observe that the
introduction of the Attention mechanism (Att), Dual-Decoder (DD) struc-
ture and Ensemble (En) method considerably improve the F1-score, Dice

20
and IoU metrics. Specifically, the dual decoder structure can improve the
F1-score, Dice, and IoU by 2.0%, 3.0%, and 2.0%, respectively. The ensem-
ble method can also enhance the F1-score, Dice, and IoU by 1.0%, 2.0%, and
2.0% respectively compared to DAUnet++.
Both visualization results and test metrics demonstrate that the pro-
posed approach can capture semantic information from both the bone and
metastatic bone regions, and outperforms other comparison models for BM
segmentation in complex situations, where multi-size lesion coexist and the
small-sized lesions are prone to being missed due to the similarity in appear-
ance.
Although our research demonstrated the advantages and prospects of us-
ing deep learning for BM segmentation from the CT-scan, still there is need
for more work to be done for BM analysis. It should be noted that our
BM-Seg dataset was collected from one center, which is UHC (University
hospital center Hedi Chaker, Sfax, Tunisia), where patients mainly come
from the south of Tunisia. Pathological data can diverge widely in clinical
scenarios from different regions and races. Moreover, larger datasets can help
in creating real scenario application for BM analysis. However, due to patient
scarcity and privacy issues, building a large dataset is very challenging. For-
tunately, our work has attracted more hospitals to join our future project in
the next BM-Seg version. Our aim is to enlarge the current BM-Seg dataset
to include BM recognition, localization and segmentation.

7. Conclusion
This paper proposes BM-Seg, the first publicly available annotated dataset
to automate the segmentation of metastatic lesions in bone CT-scan images.
We hope that this benchmark will help the community achieve more progress
on the BM segmentation problem. Moreover, we proposed EDAUnet++, an
ensemble method, which highlights areas of high uptake in CT-scan images,
to improve segmentation performance. The proposed approach outperforms
the popular cutting-edge models evaluated in our experiments and improves
significantly the F1-score.
In our future work, we will extend BMseg to create more diverse BM
dataset from multi-center, which will allow to study the generalization ability
of the machine learning algorithms to imitate the real scenario application.

21
Acknowledgement
The authors would like to express their deepest gratitude to Doctors
Zaineb Mnif and Hela Fendri for their helpful advice on various technical
issues examined in this paper, and to the Tunisian Ministry of Health for
giving permission to conduct this study.

This work was supported by The doctoral school ’sciences & technologies’
of the National School of Engineers of Sfax (ENIS) and the Computer and
Embedded Systems (CES) laboratory.

References
Allah, A.M.G., Sarhan, A.M., Elshennawy, N.M., 2023. Edge u-net:
Brain tumor segmentation using MRI based on deep u-net model with
boundary information. Expert Systems with Applications 213, 118833.
URL: https://doi.org/10.1016%2Fj.eswa.2022.118833, doi:10.1016/
j.eswa.2022.118833.

Apiparakoon, T., Rakratchatakul, N., Chantadisai, M., Vutrapongwatana,

U., Kingpetch, K., Sirisalipoch, S., Rakvongthai, Y., Chaiwatanarat, T.,
Chuangsuwanich, E., 2020. Malignet: semisupervised learning for bone
lesion instance segmentation using bone scintigraphy. Ieee Access 8, 27047–
27066.

Aslantas, A., Dandil, E., Salam, S., Çakirolu, M., et al., 2016. Cadboss: A
computer-aided diagnosis system for whole-body bone scintigraphy scans.
Journal of Cancer Research and Therapeutics 12, 787.

Bougourzi, F., Distante, C., Ouafi, A., Dornaika, F., Hadid, A., Taleb-
Ahmed, A., 2021. Per-COVID-19: A Benchmark Dataset for COVID-
19 Percentage Estimation from CT-Scans. Journal of Imaging 7, 189.
doi:10.3390/jimaging7090189. number: 9 Publisher: Multidisciplinary
Digital Publishing Institute.

Bougourzi, F., Dornaika, F., Barrena, N., Distante, C., Taleb-Ahmed, A.,
2022. CNN based facial aesthetics analysis through dynamic robust losses
and ensemble regression. Applied Intelligence URL: https://doi.org/
10.1007/s10489-022-03943-0, doi:10.1007/s10489-022-03943-0.

22
Bougourzi, F., Dornaika, F., Mokrani, K., Taleb-Ahmed, A., Ruichek, Y.,
2020. Fusing transformed deep and shallow features (ftds) for image-
based facial expression recognition. Expert Systems with Applications
156, 113459.

Cao, Y., Liu, L., Chen, X., Man, Z., Lin, Q., Zeng, X., Huang, X.,
2023. Segmentation of lung cancer-caused metastatic lesions in bone
scan images using self-defined model with deep supervision. Biomedi-
cal Signal Processing and Control 79, 104068. URL: https://doi.org/
10.1016/j.bspc.2022.104068%2Fj.compbiomed.2020.103767, doi:10.
1016/j.compbiomed.2020.103767.

Chakraborty, S., Mali, K., 2021. Sufmofpa: A superpixel and meta-heuristic

based fuzzy image segmentation approach to explicate covid-19 radiological
images. Expert Systems with Applications 167, 114142.

Cheng, D.C., Liu, C.C., Hsieh, T.C., Yen, K.Y., Kao, C.H., 2021.
Bone metastasis detection in the chest and pelvis from a whole-
body bone scan using deep learning and a small dataset. Electronics
10, 1201. URL: https://doi.org/10.3390%2Felectronics10101201,
doi:10.3390/electronics10101201.

Chmelik, J., Jakubicek, R., Walek, P., Jan, J., Ourednicek, P., Lam-
bert, L., Amadori, E., Gavelli, G., 2018. Deep convolutional neu-
ral network-based segmentation and classification of difficult to define
metastatic spinal lesions in 3d CT data. Medical Image Analysis 49, 76–
88. URL: https://doi.org/10.1016%2Fj.media.2018.07.008, doi:10.
1016/j.media.2018.07.008.

Coleman, R.E., Brown, J., Holen, I., 2020. Bone metastases. Abeloff’s
Clinical Oncology , 809–830.

da Cruz, L.B., Júnior, D.A.D., Diniz, J.O.B., Silva, A.C., de Almeida, J.D.S.,
de Paiva, A.C., Gattass, M., 2022. Kidney tumor segmentation from com-
puted tomography images using DeepLabv3 2.5d model. Expert Systems
with Applications 192, 116270. URL: https://doi.org/10.1016%2Fj.
eswa.2021.116270, doi:10.1016/j.eswa.2021.116270.

Diniz, J.O.B., Ferreira, J.L., Cortes, O.A.C., Silva, A.C., de Paiva, A.C.,
2022. An automatic approach for heart segmentation in CT scans through

23
 image processing techniques and concat-u-net. Expert Systems with Appli-
cations 196, 116632. URL: https://doi.org/10.1016%2Fj.eswa.2022.
116632, doi:10.1016/j.eswa.2022.116632.

Fan, X., Zhang, X., Zhang, Z., Jiang, Y., 2021. Deep learning-based identi-
fication of spinal metastasis in lung cancer using spectral ct images. Sci-
entific Programming 2021.

Guo, Y., Lin, Q., Zhao, S., Li, T., Cao, Y., Man, Z., Zeng, X., 2022. Au-
tomated detection of lung cancer-caused metastasis by classifying scinti-
graphic images using convolutional neural network with residual connec-
tion and hybrid attention mechanism. Insights into Imaging 13, 1–13.

Gurkan, G., Sarikaya, I., Sarikaya, A., 2019. Semiquantitative assessment

of osteoblastic, osteolytic, and mixed lytic-sclerotic bone lesions on fluo-
rodeoxyglucose positron emission tomography/computed tomography and
bone scintigraphy. World Journal of Nuclear Medicine 18, 132–136.

Hammon, M., Dankerl, P., Tsymbal, A., Wels, M., Kelm, M., May, M.,
Suehling, M., Uder, M., Cavallaro, A., 2013. Automatic detection of
lytic and blastic thoracolumbar spine metastases on computed tomogra-
phy. European Radiology 23, 1862–1870. URL: https://doi.org/10.
1007%2Fs00330-013-2774-5, doi:10.1007/s00330-013-2774-5.

Han, S., Oh, J.S., Lee, J.J., 2021. Diagnostic performance of deep learning
models for detecting bone metastasis on whole-body bone scan in prostate
cancer. European Journal of Nuclear Medicine and Molecular Imaging
49, 585–595. URL: https://doi.org/10.1007%2Fs00259-021-05481-2,
doi:10.1007/s00259-021-05481-2.

Heindel, W., Gübitz, R., Vieth, V., Weckesser, M., Schober, O., Schäfers, M.,
2014. The diagnostic imaging of bone metastases. Deutsches Ärzteblatt
International 111, 741.

Hiramatsu, Y., Hotta, K., Imanishi, A., Matsuda, M., Terai, K., 2018. Cell
image segmentation by integrating multiple CNNs, in: 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition Workshops
(CVPRW), IEEE. URL: https://doi.org/10.1109%2Fcvprw.2018.
00296, doi:10.1109/cvprw.2018.00296.

24
Jadon, S., 2020. A survey of loss functions for semantic segmentation, in:
2020 IEEE Conference on Computational Intelligence in Bioinformatics
and Computational Biology (CIBCB), IEEE. pp. 1–7.

Lei, X., Yu, X., Chi, J., Wang, Y., Zhang, J., Wu, C., 2021. Brain tumor
segmentation in MR images using a sparse constrained level set algorithm.
Expert Systems with Applications 168, 114262. URL: https://doi.org/
10.1016%2Fj.eswa.2020.114262, doi:10.1016/j.eswa.2020.114262.

Li, C., Tan, Y., Chen, W., Luo, X., Gao, Y., Jia, X., Wang, Z., 2020. Atten-
tion unet++: A nested attention-aware u-net for liver ct image segmenta-
tion, in: 2020 IEEE International Conference on Image Processing (ICIP),
IEEE. pp. 345–349.

Lin, Q., Luo, M., Gao, R., Li, T., Man, Z., Cao, Y., Wang, H., 2020.
Deep learning based automatic segmentation of metastasis hotspots in
thorax bone SPECT images. PLOS ONE 15, e0243253. URL: https:
//doi.org/10.1371%2Fjournal.pone.0243253, doi:10.1371/journal.
pone.0243253.

Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian,
M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I., 2017. A survey
on deep learning in medical image analysis. Medical image analysis 42,
60–88.

Macedo, F., Ladeira, K., Pinho, F., Saraiva, N., Bonito, N., Pinto, L.,
Gonçalves, F., 2017. Bone metastases: an overview. Oncology reviews
11.

Masoudi, S., Mehralivand, S., Harmon, S.A., Lay, N., Lindenberg, L., Mena,
E., Pinto, P.A., Citrin, D.E., Gulley, J.L., Wood, B.J., et al., 2021. Deep
learning based staging of bone lesions from computed tomography scans.
IEEE Access 9, 87531–87542.

Medixant, 2009. Radiant dicom viewer. https://www.radiantviewer.com/.

Moreau, N., Rousseau, C., Fourcade, C., Santini, G., Ferrer, L., Lacombe,
M., Guillerminet, C., Campone, M., Colombié, M., Rubeaux, M., et al.,
2020. Deep learning approaches for bone and bone lesion segmentation
on 18fdg pet/ct imaging in the context of metastatic breast cancer, in:

25
 2020 42nd Annual International Conference of the IEEE Engineering in
Medicine & Biology Society (EMBC), IEEE. pp. 1532–1535.

Noguchi, S., Nishio, M., Sakamoto, R., Yakami, M., Fujimoto, K., Emoto,
Y., Kubo, T., Iizuka, Y., Nakagomi, K., Miyasa, K., et al., 2022.
Deep learning–based algorithm improved radiologists performance in bone
metastases detection on ct. European Radiology , 1–12.

Noguchi, S., Nishio, M., Yakami, M., Nakagomi, K., Togashi, K., 2020. Bone
segmentation on whole-body CT using convolutional neural network with
novel data augmentation techniques. Computers in Biology and Medicine
121, 103767. URL: https://doi.org/10.1016%2Fj.compbiomed.2020.
103767, doi:10.1016/j.compbiomed.2020.103767.

Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K.,
Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., et al., 2018. At-
tention u-net: Learning where to look for the pancreas. arXiv preprint
arXiv:1804.03999 .

O’Sullivan, G.J., 2015. Imaging of bone metastasis: An update. World

Journal of Radiology 7, 202. URL: https://doi.org/10.4329%2Fwjr.
v7.i8.202, doi:10.4329/wjr.v7.i8.202.

Papandrianos, N., Papageorgiou, E., Anagnostis, A., Feleki, A., 2020a. A

deep-learning approach for diagnosis of metastatic breast cancer in bones
from whole-body scans. Applied Sciences 10, 997. URL: https://doi.
org/10.3390%2Fapp10030997, doi:10.3390/app10030997.

Papandrianos, N., Papageorgiou, E., Anagnostis, A., Papageorgiou, K.,

2020b. Bone metastasis classification using whole body images from
prostate cancer patients based on convolutional neural networks applica-
tion. PloS one 15, e0237213.

Papandrianos, N., Papageorgiou, E.I., Anagnostis, A., 2020c. Development

of convolutional neural networks to identify bone metastasis for prostate
cancer patients in bone scintigraphy. Annals of Nuclear Medicine 34, 824–
832. URL: https://doi.org/10.1007%2Fs12149-020-01510-6, doi:10.
1007/s12149-020-01510-6.

26
Pi, Y., Zhao, Z., Xiang, Y., Li, Y., Cai, H., Yi, Z., 2020. Auto-
mated diagnosis of bone metastasis based on multi-view bone scans us-
ing attention-augmented deep neural networks. Medical Image Analysis
65, 101784. URL: https://doi.org/10.1016%2Fj.media.2020.101784,
doi:10.1016/j.media.2020.101784.

Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convolutional networks
for biomedical image segmentation, in: International Conference on Med-
ical image computing and computer-assisted intervention, Springer. pp.
234–241.

Seyfried, T.N., Huysentruyt, L.C., 2013. On the origin of cancer metastasis.

Critical Reviews in Oncogenesis 18.

Shimizu, A., Wakabayashi, H., Kanamori, T., Saito, A., Nishikawa, K.,
Daisaki, H., Higashiyama, S., Kawabe, J., 2020. Automated measurement
of bone scan index from a whole-body bone scintigram. International Jour-
nal of Computer Assisted Radiology and Surgery 15, 389–400.

Steeg, P.S., 2006. Tumor metastasis: mechanistic insights and clinical chal-
lenges. Nature medicine 12, 895–904.

Turpin, A., Girard, E., Baillet, C., Pasquier, D., Olivier, J., Villers, A.,
Puech, P., Penel, N., 2020. Imaging for metastasis in prostate cancer: a
review of the literature. Frontiers in Oncology 10, 55.

Vandemark, R.M., Shpall, E.J., Affronti, M.L., 1992. Bone metastases

from breast cancer. Journal of Computer Assisted Tomography 16, 608–
614. URL: https://doi.org/10.1097%2F00004728-199207000-00022,
doi:10.1097/00004728-199207000-00022.

Vantaggiato, E., Paladini, E., Bougourzi, F., Distante, C., Hadid, A., Taleb-
Ahmed, A., 2021. COVID-19 Recognition Using Ensemble-CNNs in Two
New Chest X-ray Databases. Sensors 21. URL: https://www.mdpi.com/
1424-8220/21/5/1742, doi:10.3390/s21051742.

Zeiss, 2021. Apeer platform. https://www.apeer.com/.

Zhang, J., Huang, M., Deng, T., Cao, Y., Lin, Q., 2021. Bone metastasis
segmentation based on improved u-net algorithm, in: Journal of Physics:
Conference Series, IOP Publishing. p. 012027.

27
 Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J., 2018. Unet++:
A nested u-net architecture for medical image segmentation, in: Deep
learning in medical image analysis and multimodal learning for clinical
decision support. Springer, pp. 3–11.

28

View publication stats