Professional Documents
Culture Documents
Heliyon
journal homepage: www.cell.com/heliyon
Research article
A R T I C L E I N F O A B S T R A C T
Keywords: Diabetic retinopathy is not just the most common complication of diabetes but also the leading cause of adult
Retinal vessel segmentation blindness. Currently, doctors determine the cause of diabetic retinopathy primarily by diagnosing fundus images.
Deep learning Large-scale manual screening is difficult to achieve for retinal health screen. In this paper, we proposed an
U-net
improved U-net network for segmenting retinal vessels. Firstly, due to the lack of retinal data, pre-processing of
Bi-FPN
the raw data is required. The data processed by grayscale transformation, normalization, CLAHE, gamma trans-
formation. Data augmentation can prevent overfitting in the training process. Secondly, the basic network
structure model U-net is built, and the Bi-FPN network is fused based on U-net. Datasets from a public challenge
are used to evaluate the performance of the proposed method, which is able to detect vessel SP of 0.8604, SE of
0.9767, ACC of 0.9651, and AUC of 0.9787.
* Corresponding author.
E-mail address: k.ren@njust.edu.cn (K. Ren).
https://doi.org/10.1016/j.heliyon.2022.e11187
Received 9 March 2022; Received in revised form 4 August 2022; Accepted 17 October 2022
2405-8440/© 2022 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
K. Ren et al. Heliyon 8 (2022) e11187
the features at the bottom layer with the features at the top layer, thus
Table 1. Segmentation results of different algorithms. improving the accuracy of accurate vessel segmentation. The results of
Type Methods SE SP ACC AUC our experiments demonstrate that the network achieves better perfor-
Unsupervised learning Zana [13] 0.6971 — 0.9377 0.8984 mance on the retinal vessel segmentation task.
AI-Diri [ [14]] 0.7282 0.9551 — —
Miri [15] 0.7352 0.9795 0.9458 —
2. Methods
Fraz [16] 0.7152 0.9795 0.9430 —
In this section, our proposed segmentation method will be described
You [17] 0.7410 0.9759 0.9434 —
in detail. The preprocessing includes grayscale transformation, normal-
Supervised learning Fraz [18] 0.7152 0.9751 0.9430 —
ization, CLAHE, and gamma transformation in the training phase. To
Marin [19] 0.7067 0.9769 0.9452 0.9588
increase the number of training samples, we use a slicing method to slice
Ricci [20] — 0.9801 0.9563 0.9558
the original image into small-sized image blocks of uniform size and
Sohini [21] 0.7249 — 0.9519 0.9620
augmented samples as the input to the network.
Oliveira [22] 0.8039 0.9804 0.9576 0.9821
Alom [23] 0.7792 0.9813 0.9556 0.9784
2.1. Pre-processing
U-NET 0.7240 0.9848 0.9516 0.9735
Tamim [27] 0.7542 0.9843 0.9607 —
Fundus images have disadvantages such as uneven brightness, poor
This article 0.8064 0.9767 0.9551 0.9787
contrast, and strong noise, requiring per-processing before input the
network for training. Figures 1(A) and 1(B) show the original fundus
image and manually labeled vascular map of the retinal vessels.
the level set-based method were used. It is generally considered The fundus images are converted to grayscale and then normalized.
that the cross-sectional shape of the blood vessel conforms to a The conversion equation is shown as follows:
Gaussian distribution. Since the model gives poor segmentation
results for low-resolution images, Gang et al. proposed an Igray ¼ 0:299 Ir þ 0:587 Ig þ 0:114 Ib (1)
improved approach based on the above model, using Gaussian Among them Ir , Ig andIb are the RGB channel components of fundus
difference for modeling. Gonzalez et al. [5] proposed a joint
images. As shown in Eq. (1), Igray is the result of processing. It is stan-
framework for not only retinal vessel but also optic disc segmen-
dardized and the standardization equation is shown in Eq. (2).
tation, which uses a graph cut method for segmenting the vessel
tree, followed by a Markov random field graph construction, and a Igray μ
Inorm ¼ (2)
compensation factor method disc segmentation was performed. σ
(2) Supervised learning refers to training to obtain an optimal model
based on a known relationship between the input and output of a Among them, μ denotes the mean value,σ denotes variance.
data set, with both features and labels on the training data. Su- Figure 2(B) is the gray scale of Figure 2(A).
pervised learning algorithms classify each pixel and determine Data augmentation can increase amount of the samples in the training
whether the pixel belongs to the vessel or the background. In set, effectively alleviating model overfitting, and bringing stronger
general, supervised methods tend to perform better than unsu- generalization ability to the model. Data augmentation is generally
pervised methods. In [6], support vector machine (SVM) classi- divided into three categories: geometric transformations, color space
fiers were proposed to be trained using manually designed transformations, and pixel point operations. Cropping is a common
features, which showed a significant performance improvement method among geometric transformations and a long, more obvious way
compared to unsupervised algorithm segmentation results. How- to enhance data, which can augment data without destroying the accu-
ever, such methods are based on hand-designed features and have racy condition. In this paper, based on this cropping property, we choose
poor generalization ability. In the research field of machine or to augment the data by random slicing.
deep learning, many excellent algorithms have shown excellent The retinal dataset used in this paper contains 20 training sets and 20
performance on publicly available datasets. In 2016, Laskowski test sets. To fit our improved network structure, the original images need
et al. [7] proposed a geometric transform-based data augmenta- to be chunked. The patches which have been segmented are merged to
tion for training CNN networks to achieve vascular segmentation. create the segmented blood vessel image at the end of the procedure.
Lahiri et al. [8] proposed an auto-encoder-based inheritance The original image 565*565 is sliced into 48*48 size patches, as
network architecture that implements vascular segmentation by shown in Figure 3, each obtained by randomly selecting its center within
combining all auto-encoder outputs. To achieve vascular seg- the image. Some patches are located in the circular field of view, some
mentation. Maji et al. [9] utilized an ensemble of 12 deep CNN patches are located outside the circular field of view, and others are
networks and averaged the outputs of all networks as the final located above the boundary. In this way, not only the dataset is enlarged,
input. Deep retinal images understood VGGNet [10] as the basis but also the network can learn how to distinguish FOV boundaries.
for effectively solving retinal vascular and optic disc segmentation
using a whole convolutional network with two specific layers. 2.2. Image enhancement
Compared with traditional algorithms, deep learning reduces human As shown in Figure 4(A)&(B), we use CLAHE (Contrast Limited
intervention, builds better prediction models, and avoid bias due to the Adaptive Histogram Equalization) for image enhancement. The adaptive
lack of data. histogram equalization algorithm expands the local contrast and displays
This paper proposes an improved U-net model for retinal vessel seg- smooth details by performing histogram equalization in a rectangular
mentation to address the shortcomings of the algorithms mentioned area around the currently processed pixel. Fortunately, CLAHE can
above. Firstly, the data are pre-processed, and CLAHE performs image effectively limit the noise amplification generated in the process of
enhancement, and then the enhanced image is sliced. The processed data contrast expansion. It mainly contains image chunking, block linear
are fed into the network based on the U-net model and fused with the Bi- interpolation, and layer filtering, and blending operations with the
FPN fusion network in the Efficient Det network. The fused network fuses original image.
2
K. Ren et al. Heliyon 8 (2022) e11187
Figure 1. Original fundus image (A) and Manually labeled vascular map of the retinal vessels (B).
2.3. U-net performance efficiently. Low-level features have better resolution and
contain more information on location and detail. High-level features
The U-net consists of an input layer, a hidden layer, and an output have low resolution and poor perception of details. The way to efficiently
layer. Its structure is shown in Figure 5. The hidden layer can be divided fuse the features from different levels is the key to optimize the seg-
into an up-sampling and a down-sampling part, distinguishing decoders mentation model.
and encoders. Down-sampling consists of convolution layer and pooling Model efficiency is crucial in computer vision. Bi-FPN is proposed to
layer, which play the role of path shrinking in the network and capture pursue a more efficient multi-scale fusion method. Bi-FPN is used to
global information. The up-sampling consists of convolutional and replace the skip connection, and the concept of weight is proposed to
deconvolution layers, which play the role of path expansion in the balance the feature information of various scales better and improve the
network and locate pixel points. The output layer is an end-to-end efficiency of the model. The higher level of semantic information is more
network that classifies each pixel point of the feature map with the able to help us segment the target accurately. Some of the fine target
same size as the original image after up-sampling by the softmax [12] information may be ignored when doing down-sampling operation in
activation function. That is, the input image is of the same size as the deep images. The proposed multi-scale feature fusion network is a good
output image. solution to this problem.
P ωi
O ¼ i εþP I , we are implementing a ReLU procedure after every
ω i
j j
3
K. Ren et al. Heliyon 8 (2022) e11187
ω1 P6 in þ ω2 ResizeðP7 inÞ amplified and may reduce the speed of network convergence. In order to
P6 td ¼ Conv (3)
ω1 þ ω2 þ ε defeat this shortcoming, BN layer was implemented before using ReLU
procedure [28]. BN layer can standardize and evenly distribute the
0 0 0
ω P6 in þ ω2 P6 td þ ω3 ResizeðP5 outÞ network layer data, and these new data will not have a significant impact
P6 out ¼ Conv 1 (4) on subsequent layers, which can improve the convergence efficiency of
ω1 þ ω2 þ ω3 þ ε
0 0 0
4
K. Ren et al. Heliyon 8 (2022) e11187
5
K. Ren et al. Heliyon 8 (2022) e11187
Figure 8. Evaluation index: ROC curve (A) and Precision-Recall curve (B).
3. Experiments slicing, the size of the images is 48*48 pixels, and the information of its
color space is retained, and the size is 48*48*3.
3.1. Datasets In this paper, the categorical cross-entropy function is used as the loss
function. As shown in Eq. (5), the cross-entropy evaluates how the
In this paper, we use the DRIVE [11] dataset. Images in the DRIVE probability distribution obtained from the current training differs from
dataset were obtained from the Dutch Diabetic Retinal Screening Pro- the actual distribution.
gram, using fundus images obtained with a 45-degree field of view (FOV)
Canon CR5 non-dilated 3CCD camera. The size of each fundus image is X
n
loss ¼ yc c
i1 log yi1 þ y c
i2 log yi2 þ ::: þ y im log yim (5)
768 584 8-bit color images. The FOV of each image in the dataset is i¼1
circular with a diameter of 540 pixels, and the corresponding FOV image
is given as a masked image. Among them, 33 are standard retinal images,
3.3. Evaluation methodology
7 have mild lesions, and each image has a gold standard segmented by
hand by experts.
Experiments were conducted to analyze and compare the perfor-
mance of the segmentation algorithm proposed in this paper with other
3.2. Experimental process
algorithms using evaluation indexes such as accuracy, sensitivity, speci-
ficity, precision, and AUC, which are defined as follows:
In model training, 20 images of the training set are sliced and divided
into 19000 patches, and 90% of them are selected for training in each TP þ TN
training round, and the remaining 10% are used for validation. After ACC ¼ (6)
TP þ TN þ FP þ FN
6
K. Ren et al. Heliyon 8 (2022) e11187
4. Conclusion
Additional information
7
K. Ren et al. Heliyon 8 (2022) e11187
[2] S. Chaudhuri, S. Chatterjee, N. Katz, et al., Detection of blood vessels in retinal [16] M.M. Fraz, S.A. Barman, P. Remagnino, et al., An approach to localize the retinal
images using two-dimensional matcher filters[J], IEEE Trans. Med. Imag. 8 (3) blood vessels using bit planes and centerline detection[J], Comput. Methods Progr.
(1989) 263–269. Biomed. 108 (2) (2012) 600–616.
[3] G. Azzopardi, N. Strisciuglio, M. Vento, et al., Traniable COSFIRE filters for vessel [17] X.G. You, Q.M. Peng, Y. Yuan, et al., Segmentation of retinal blood vessels using the
delineation with application in retinal images[J], Med. Image Anal. 19 (1) (2015) radial projection and semi-supervised approach[J], Pattern Recogn. 44 (10-11)
46–57. (2011) 2314–2324.
[4] J. Zhang, B. Dashtbozorg, E. Bekkers, Robust Retinal Vessel Segmentation via [18] M.M. Fraz, P. Remagnino, A. Hoppe, et al., Blood vessel segmentation
Locally Adaptive Derivative Frames in Orientation Scores[J], IEEE Transactions on methodologies in retinal images: a survey[J], Comput. Methods Progr. Biomed. 108
Medical Imaging 35 (12) (2016) 2631–2644. (1) (2012) 407–433.
[5] A. Salazar-Gonzalez, D. Kaba, Y. Li, Segmentation of blood vessels and optic disc in [19] D. Marin, M.E. Gegundez-Arias, B. Ponte, et al., An exudate detection method
retinal images[J], IEEE Journal of Bio-medical and Health Informatics 18 (6) (2014) for diagnosis risk of diabetic macular edema in retinal images using feature-
1874–1886. based and supervised classification[J], Med. Biol. Eng. Comput. 56 (8) (2018)
[6] M.M. Fraz, S.A. Barman, P. Remagnino, et al., An approach to localize the retinal 1379–1390.
blood vessels using bit planes and centerline detection[J], Comput. Methods Progr. [20] E. Ricci, R. Perfetti, Retinal blood vessel segmentation using line operators and
Biomed. 108 (2) (2012) 600–616. support vector classification[J], IEEE Trans. Med. Imag. 26 (10) (2007) 1357–1365.
[7] D. Marin, M.E. Gegundez-Arias, B. Ponte, et al., An exudate detection method for [21] R. Sohini, D.D. Koozekanani, K.K. Parhi, Blood vessel segmentation of fundus
diagnosis risk of diabetic macular edema in retinal images using feature-based and images by major vessel extraction and subimage classification[J], IEEE Journal of
supervised classification[J], Med. Biol. Eng. Comput. 56 (8) (2018) 1379–1390. Biomedical and Health Informatics 19 (3) (2015) 1118–1128.
[8] E. Ricci, R. Perfetti, Retinal blood vessel segmentation using line operators and [22] A. Oliveira, S. Pereira, C.A. Silva, Retinal vessel segmentation based on fully
support vector classification[J], IEEE Trans. Med. Imag. 26 (10) (2007) 1357–1365. convolutional neural networks[J], Expert Syst. Appl. 112 (2018) 229–242.
[9] R. Sohini, D.D. Koozekanani, K.K. Parhi, Blood vessel segmentation of fundus [23] Alom M Z, Hasan M, Yakopcic C, et al. Recurrent Residual Convolutional Neural
images by major vessel extraction and subimage classification[J], IEEE Journal of Network Based on U-Net (R2U-Net) for medical image segmentation [DB/OL].
Biomedical and Health Informatics 19 (3) (2015) 1118–1128. [24] Francois Chollet, Xception: Deep Learning with Depthwise Separable Convolutions.
[10] A. Oliveira, S. Pereira, C.A. Silva, Retinal vessel segmentation based on fully CVPR, 2017, pp. 1610–2357.
convolutional neural networks [J], Expert Syst. Appl. 112 (2018) 229–242. [25] Sifre Laurent, Rigid-motion scattering for image classification, Ph.D. thesis section 6
[11] J. Staal, M.D. Abramoff, NIEMEIJER, et al., Ridge-based vessel segmentation in (2) (2014).
color images of the retinal[J], IEEE Trans. Med. Imag. 23 (4) (2004) 501–509. [26] M. Tan, R. Pang, Q.V. Le, EfficientDet: scalable and efficient object detection, IEEE/
[12] A.A. Nahid, M.A. Mehrabi, Y.N. Kong, Histopathoiogical Breast Cancer Image CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Classification by Deep Neural Network Techniques Guided by Local clustering[J], 10778–10787.
2018, BidMed Research International, 2018, p. #2362108. [27] N. Tamim, G. Azim, H. Nassar, Retinal blood vessel segmentation using hybrid
[13] F. Zana, J.C. Klein, Segmentation of vessel-like petterns using mathematical features and multi-layer perceptron neural networks, Symmetry 12 (2020) 894.
morphology and curvature evaluation[J], IEEE Trans. Image Process. 10 (7) (2001) [28] S. Ioffe, C. Szegedy, Batch normalization:accelerating deep network training by
1010–1019. reducing internal covarite shift, Lille, France, ICML 2015 Proceedings of the 32nd
[14] B. Al-Diri, A. Hunter, D. Steel, An active contour model for segmenting and International Conference on Machine Learning – Volume 37 (2015) 448–456.
measuring retinal vessels[J], IEEE Trans. Med. Imag. 28 (9) (2009) 1488–1497. [29] Y. Cai, Y. Li, X. Gao, Y. Guo, Retinal vessel segmentation method based on improved
[15] M.S. Miri, A. Mahloojifar, Retinal image analysis using curvelet transform and deep U-net, in: Z. Sun, R. He, J. Feng, S. Shan, Z. Guo (Eds.), Biometric Recognition.
multistructure elements morphology by reconstruction[J], IEEE (Inst. Electr. CCBR 2019. Lecture Notes in Computer Science 11818 (2019) 321–328. Springer,
Electron. Eng.) Trans. Biomed. Eng. 58 (5) (2011) 1183–1192. Cham.