Professional Documents
Culture Documents
Chromosome ICCE 2022
Chromosome ICCE 2022
Abstract—Karyotyping is critical for screening genetic diseases (XX for female and XY for male). Suppose there is a structural
in an early stage. However, the manual karyotyping process is or numerical anomaly of the chromosomes that could be
a labor-intensive task. This paper focuses on automatic chro- evidence of genetic disorders [3]. Fig. 1 is an example of
mosome image classification, which is a step in karyotyping.
We propose a convolutional neural network architecture for a metaphase image and its corresponding karyogram of a
efficient type and polarity classification of the chromosome image, Trisomy 13 patient.
namely ETPC, which is leveraged from the EfficientNet family’s
development. The ETPC’s classifier with a weighted classification
loss function are designed for efficiency in the training process.
We perform our experiment with two training scenarios on four
clinical datasets. The experiment on a dataset of 28,225 original
chromosome images demonstrates that the proposed network
achieved comparable results, with an accuracy of 95.3% for the
type classification and 99% for the polarity classification, while
having a significantly smaller number of parameters than state-
of-the-art methods. Furthermore, the experiment on multiple
datasets shows that the proposed network can dramatically
improve the classification accuracy on new datasets with a small
amount of fine-tuning data. (a) (b)
Index Terms—Karyotyping, chromosome, classification, deep
learning, EfficientNetV2 Fig. 1. An example of a metaphase image taken by microscope (a) and the
corresponding karyogram (b) of a Trisomy 13 patient with an extra copy of
I. I NTRODUCTION chromosome 13 (the red arrow).
Human chromosomes provide information in genetic disor- The procedure of manual karyotyping often requires a
ders, supporting cytogeneticists in screening and diagnosing meticulous effort from experienced cytogeneticists, which
diseases such as trisomy, deletions, and other genetic symp- is tediousness, low-reproducibility, and time-consuming. In
toms at an early stage [1]. In clinical practice, karyotyping, clinical practice, cytogeneticists may diagnose up to thirty
a process of separating, classifying, and ordering the chro- metaphase images to draw a reliable conclusion [2]. With the
mosome images in a cultured cell taken by a microscope, motivation of reducing the workload for cytogeneticists, this
acts a crucial role in diagnosing chromosome abnormalities paper will focus on automating the chromosome classification
[2]. Standard karyotyping uses the chromosome images at the since it is an essential and labor-intensive task in the karyotyp-
metaphase stage of mitosis (metaphase image) since all chro- ing process. Some challenges of the chromosome classification
mosomes become distinguishable by their unmistakable struc- can be listed as follows: 1) The chromosomes exhibit a
ture. Cytogeneticists use the characteristics of chromosomes to non-rigid intrinsic nature that makes the chromosomes have
classify and locate them in the table-like format known as the entirely different shapes and orientations; 2) In a metaphase
karyogram. An individual human has 46 chromosomes divided image, the chromosomes can be in random positions and
into 22 pairs of autosomes and one pair of sex chromosomes randomly rotated, and thus classification of polarity (showing
p-arm is upward or downward) is essential in performing methods, which are inherited from AlexNet. The study yielded
karyotyping [4]; 3) Differences in the cell cultivating processes an accuracy in chromosome type classification of 92.5% and
and imaging systems in medical centers lead to diversity 93.8%, respectively.
among the chromosome images. Many studies that used variations of deeper CNN and
The purpose of this study is to propose and access an leveraged the efficiency of the learning process have improved
automatic method for the type and polarity classification of the the accuracy of the classification task. Lin et al. (2020)
chromosome images. Our approach is to use a convolutional [6] proposed an augmentation technique, namely CDA, and
neural network (CNN) for chromosome classification with the used Inception-ResNet for the chromosome type classification.
aim for fast convergence, having a low number of parameters The CDA showed a practical improvement in classification
while still achieving high accuracy. accuracy from 87.5% to 96%. In the further study [7], MixNet
The main contributions of the paper are listed as follows: and a training algorithm are introduced and achieved the
• We propose a new CNN architecture for the purpose of accuracy of 98.7% and 96.5% on the G-band data and Q-
the chromosome type and polarity classification, with a band data, respectively. Al-Kharraz et al. (2021) [5] ensemble
weighted classification loss function for efficiency during a decision of three CNNs, including VGG19, ResNet50,
training its model, and investigate the optimal weight and MobileNetV2. The network uses average voting for the
coefficient for the classification task. chromosome type classification and yielded an accuracy of
• We perform experiments and evaluations for chromosome 97%.
classification using multiple datasets from several hospi-
tals. B. Multi-task for chromosome classification
The rest of this paper is organized as follows: The following Using classical image processing methods, studies of chro-
section reviews previous studies for chromosome classifica- mosome classification collect multi-information from chromo-
tion. Section III provides information in detail on the methods some images, bringing high efficiency to the classification
used in this study. The experimental results are presented in process. Wang et al. [1] proposed a multi-stage rule-based
section IV. We place the discussion in section V. Finally, we scheme for identifying the centromere and assigning the polar-
conclude the findings from this study in section VI. ity of chromosomes. The two pieces of extracted information
II. R ELATED W ORK help classify the chromosomes into three groups: metacentric,
submetacentric, and acrocentric. Poletti et al. [14] presented
A. Chromosome type classification a modular framework for the chromosome type and polarity
Several methods for chromosome classification have been classification, which relies on hand-craft feature extraction,
published recently [4]–[7], which can be divided into two cat- including the medial axis and various feature vectors.
egories: classical methods and deep learning-based methods. Deep learning techniques are proposed in the literature,
The pipeline of classical methods often contained two steps: showing the power of multi-task learning [3], [4]. Although
the first step is handcraft features extraction from the chro- polarity classification has implications in clinical medical
mosome images, and the second step is feeding these features procedures, only a few studies have leveraged the power of
to the classifier (e.g., multi-layer perceptron, support vector deep learning techniques to produce this information. Qin et
machine. . . ) to determine the type of chromosome [8], [9]. al. (2019) [4] proposed Varifocal-Net for chromosome type
Markou et al. [8] proposed a robust method that uses features and polarity classification. The Varifocal-Net composed of two
based on the medial axis, and a hybrid 2-level classifier is networks, G-Net and L-Net. The G-Net extracts the global
used for human chromosome type classification. Oskouei et features and then detects the region to zoom into the more
al. [9] combined the wavelet into a neural network that uses delicate part in the chromosome image; the zoomed part is
various density-based features to classify the chromosomes in used to extract local features by L-Net. After the features
group E (chromosomes 16 to 18). The limitation of classical extraction stage, two multi-layer perceptron classifiers are used
methods is that authors must carefully design suitable features to exploit two scales features for boosting the performance of
for a specific dataset. the classification results. The study reported the best accuracy
Applying deep learning techniques for medical image anal- of type and polarity classification are 98.7% and 99.2%,
ysis has garnered the attention of researchers in recent years. respectively. Zhang et al. (2021) [3] presented a CNN for
Deep learning techniques for the chromosome classification extracting multi-resolution features. The extracted features are
proposed in the literature have been outperforming the clas- fused and fed in two multi-layer perceptron classifiers to
sical methods [10]. The convolutional neural network is a classify the chromosomes type and polarity. The networks
class of deep learning techniques recently popular in the achieved accuracies of 98.1% and 99.8% for the type and
chromosome classification fields. Jindal et al. (2017) [11] pre- polarity classification, respectively.
sented a Siamese network for chromosomes type classification, Although several CNN-based chromosome classification
composed of two CNNs to handle the limit data problem. methods have been already published, there is a lack of a study
The study reported the best accuracy of 85.6%. Zhang et al. on the evaluation of the performance on multiple datasets.
(2018) [12] and Hu et al. (2019) [13] utilized CNN-based Therefore, the purpose of the study is also to primarily evaluate
the power of CNNs on multiple chromosome datasets from Similarly, the loss value of polarity classification task be-
several hospitals. tween output vector op and ground truth vector gp is calculated
by:
III. M ETHOD 1
1X p
The proposed CNN architecture, named ETPC, is illustrated Lp (op , gp ) = − (g log(opi ) + (1 − gip ) log(1 − opi ))
2 i=0 i
in Fig. 2. The CNN architecture consists two-phase: 1) Fea-
tures extraction that reuses the EfficientNetV2 backbone. 2) (2)
The classifier is designed to classify the chromosome type The loss function is used for training the ETPC is given by:
and polarity as a multi-task learning task. This section will
present details on ETPC architecture, the classifier, and the Ltotal = λLt (ot , gt ) + (1 − λ)Lp (op , gp ) (3)
weighted loss function for the training process.
where λ is the weighted coefficient for controlling the balanc-
A. Features extraction using EfficientNetV2 backbone ing between the two loss terms. The optimal λ coefficient is
determined in the experimental section.
The EfficientNet family was first introduced by Tan and
Le (2020) [15], which achieved state-of-the-art results on the IV. E XPERIMENT AND E VALUATION
ImageNet dataset, while the computational cost is significantly
A. Clinical data and preprocessing
reduced compared to previously published CNNs. The fun-
damental ideal of EfficientNet is compound model scaling to The data used in this study were collected from four
solve an optimization problem related to predefined parameters medical institutions with 886 karyograms with a total of 40869
in a baseline model. The authors recently have continued chromosome images. Information about the number of images
developing a new architecture named EfficientNetV2 [16]. The used for the training and testing phase is listed in Table I.
latest version model reduces training time and has smaller The first dataset, Pki dataset, was collected by the University
parameters while yielding comparable accuracy since this of Passau, Germany, and published by Ritter et al. [18],
architecture could generate various models using compound including 611 G-bands karyograms. We use 61 karyograms
model scaling techniques. (including 48 abnormal karyograms and 13 normal karyo-
To take the advantage of EfficientNetV2, we chose grams) that have been classified from our previous study [10].
EfficientNetV2-B2 as the backbone of ETPC to extract chro- We divide the remaining 80% of the images for training ETPC
mosome images’ features, since EfficientNetV2-B2 compro- and use 20% to experiment to find the optimal loss function
mises between the number of parameters and classification coefficient and select the best training model for ETPC.
accuracy. Fig. 2 shows the EfficientNetV2-B2 backbone in the The second dataset was published by Hu et al. [13] (Lu-
features extraction part, including seven stages for high-level daopei dataset). The data includes the chromosomes images
features extraction. of 91 karyograms collected at Beijing Ludaopei Hospital.
The chromosome images have been expertly labeled and
B. Mutil-task learning for chromosome type and polarity cropped into individual chromosome images with dimensions
classification of 100x220 pixels in width and height, respectively.
The third dataset is a public chromosome benchmark
High-level features are extracted via the ETPC’s backbone
dataset, collected by the Italy University of Padova (Biomlab
and then fed into the classifier stage (Fig. 2). Different from
dataset). Biomlab dataset includes 119 Q-bands karyograms.
the original classifier of EfficientNetV2-B2, the classifier stage
This dataset is widely used in chromosome classification
contains an average pooling layer followed by two fully
studies [5], [7], [11]. The chromosome images have been
connected layers to get the prediction vectors of type and
expertly labeled and cropped to multiple sizes based on the
polarity of the chromosome images. The previous studies on
shape of chromosomes.
multi-task chromosome classification use the Softmax function
The final dataset (Guangdong dataset) consisting of 65 G-
combined with Cross-Entropy loss [3], [4]. Nevertheless, this
bands karyograms studied and published by Lin et al. [6], [7].
is a multi-class multi-label problem, and Gour et al. already
Data were collected from Guangdong Women and Children
showed that the Sigmoid function is suitable for this type of
Hospital.
problem [17]. Therefore, we use the Sigmoid function for the
The data listed above have different characteristics. We
last classifier layer combined with Binary Cross-Entropy Loss
preprocess by segmenting every chromosome and pad it to
for ETPC.
dimensions of 256x256 pixels. Fig. 3 displays examples of
Loss value of type classification between the output vector
chromosome 1 on four datasets after preprocessing step.
ot and ground truth vector gt is defined as:
The Ludaopei dataset has chromosome images with a 400-
23
1 X t bands resolution instead of a 600-bands resolution like the
Lt (ot , gt ) = − (g log(oti ) + (1 − git ) log(1 − oti )) Pki dataset. Biomlab dataset uses Q-stand instead of G-
24 i=0 i
stand compared to other datasets. In the Guangdong dataset,
(1) some chromosome images are cropped, causing a loss of
Stage 0-6: Features extraction Stage 7: Classifier
Fig. 2. The ETPC architecture consists of two phases; the backbone for features extraction (stage 0-6) and the modified classifier (stage 7).
Fig. 3. Examples of chromosome 1 on four datasets after preprocessing: three G-band datasets include Pki, Ludaopei, and Guangdong; Biomlab is a Q-band
dataset.
really high. This is a premise for further studies on multiple [5] M. Al-Kharraz, L. A. Elrefaei, and M. Fadel, “Classifying chromosome
datasets; otherwise, more manual corrections are required from images using ensemble convolutional neural networks,” in Applications
of Artificial Intelligence in Engineering. Springer, 2021, pp. 751–764.
cytogeneticist when using it in the clinical practice. [6] C. Lin, G. Zhao, Z. Yang, A. Yin, X. Wang, L. Guo, H. Chen,
Our study contains some limitations. First, the experiment Z. Ma, L. Zhao, H. Luo et al., “Cir-net: Automatic classification of
on Pki dataset was carried out using a limited number of human chromosome based on inception-resnet architecture,” IEEE/ACM
Transactions on Computational Biology and Bioinformatics, 2020.
chromosome images. We hypothesize that a large number of [7] C. Lin, G. Zhao, A. Yin, L. Guo, H. Chen, and L. Zhao, “Mixnet:
chromosome images used in the training phase may signifi- A better promising approach for chromosome classification based on
cantly improve the performance of ETPC. Secondly, we only aggregated residual architecture,” in 2020 International Conference on
Computer Vision, Image and Deep Learning (CVIDL). IEEE, 2020,
accomplished a pre-primary study with ten karyograms for pp. 313–318.
fine-tuning on a new dataset. A further study on finding a [8] C. Markou, C. Maramis, A. Delopoulos, C. Daiou, and A. Lam-
suitable number of images for a deep learning model that can bropoulos, “Automatic chromosome classification using support vector
machines,” in Pattern Recognition: Methods and Applications. iConcept
adapt to a new dataset may be required. Press, 2012, pp. 1–24.
[9] B. C. Oskouei and J. Shanbehzadeh, “Chromosome classification based
VI. C ONCLUSION on wavelet neural network,” in 2010 International Conference on Digital
In this study, we have proposed a CNN architecture, ETPC, Image Computing: Techniques and Applications. IEEE, 2010, pp. 605–
610.
for the chromosome type and polarity classification, one of [10] N. H. Thinh, N. H. H. Son, P. T. V. Huong, N. T. C. Nhung,
the essential tasks in automated karyotyping. Our experiment D. T. Ram, N. T. B. Minh, and L. M. Ha, “A web-based tool for
indicated that the ETPC achieved 95.3% and 99% accuracy semi-interactively karyotyping the chromosome images for analyzing
chromosome abnormalities,” in 2020 7th NAFOSTED Conference on
for type and polarity classification. In addition, ETPC achieved Information and Computer Science (NICS). IEEE, 2020, pp. 433–437.
comparable results with the state-of-the-art methods, while the [11] S. Jindal, G. Gupta, M. Yadav, M. Sharma, and L. Vig, “Siamese
number of parameters is significantly smaller than that of most networks for chromosome classification,” in Proceedings of the IEEE
international conference on computer vision workshops, 2017, pp. 72–
of the the-state-of the-art methods. Moreover, we designed the 81.
experiment to evaluate ETPC on multiple datasets. The results [12] W. Zhang, S. Song, T. Bai, Y. Zhao, F. Ma, J. Su, and L. Yu,
showed that ETPC has dramatically improved the classification “Chromosome classification with convolutional neural network based
deep learning,” in 2018 11th International Congress on Image and
accuracy with a small amount of fine-tuning data, yielding Signal Processing, BioMedical Engineering and Informatics (CISP-
a potential for an adaptation for multiple datasets of several BMEI). IEEE, 2018, pp. 1–5.
hospitals. [13] X. Hu, W. Yi, L. Jiang, S. Wu, Y. Zhang, J. Du, T. Ma, T. Wang, and
X. Wu, “Classification of metaphase chromosomes using deep convolu-
VII. ACKNOWLEDGMENTS tional neural network,” Journal of Computational Biology, vol. 26, no. 5,
pp. 473–484, 2019.
We would like to thank Assoc. Prof. Nguyen Linh Trung for [14] E. Poletti, E. Grisan, and A. Ruggeri, “A modular framework for the
the valuable discussion in this study. We would like to thank automatic classification of chromosomes in q-band images,” Computer
the anonymous reviewers for their constructive comments. methods and programs in biomedicine, vol. 105, no. 2, pp. 120–130,
2012.
R EFERENCES [15] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for con-
volutional neural networks,” in International conference on machine
[1] X. Wang, B. Zheng, S. Li, J. J. Mulvihill, and H. Liu, “A rule-based learning. PMLR, 2019, pp. 6105–6114.
computer scheme for centromere identification and polarity assignment [16] M. Tan and Q. Le, “Efficientnetv2: Smaller models and faster training,”
of metaphase chromosomes,” Computer Methods and Programs in in International Conference on Machine Learning. PMLR, 2021, pp.
Biomedicine, vol. 89, no. 1, pp. 33–42, 2008. 10 096–10 106.
[2] R. Remani Sathyan, G. Chandrasekhara Menon, R. Thampi, and J. H. [17] N. Gour and P. Khanna, “Multi-class multi-label ophthalmological
Duraisamy, “Traditional and deep-based techniques for end-to-end auto- disease detection using transfer learning based convolutional neural
mated karyotyping: A review,” Expert Systems, vol. 39, no. 3, p. e12799, network,” Biomedical Signal Processing and Control, vol. 66, p. 102329,
2022. 2021.
[3] J. Zhang, W. Hu, S. Li, Y. Wen, Y. Bao, H. Huang, C. Xu, and [18] G. Ritter and L. Gao, “Automatic segmentation of metaphase cells based
D. Qian, “Chromosome classification and straightening based on an on global context and variant analysis,” Pattern Recognition, vol. 41,
interleaved and multi-task network,” IEEE Journal of Biomedical and no. 1, pp. 38–55, 2008.
Health Informatics, vol. 25, no. 8, pp. 3240–3251, 2021. [19] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
[4] Y. Qin, J. Wen, H. Zheng, X. Huang, J. Yang, N. Song, Y.-M. Zhu,
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
L. Wu, and G.-Z. Yang, “Varifocal-net: A chromosome classification
approach using deep convolutional networks,” IEEE transactions on
medical imaging, vol. 38, no. 11, pp. 2569–2581, 2019.
[20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image and pattern recognition, 2016, pp. 770–778.
recognition,” in Proceedings of the IEEE conference on computer vision