Automatic screening of fundus images using a combination of
convolutional neural network and hand-crafted features
Balazs Harangi, Member, IEEE, Janos Toth, Member, IEEE, Agnes Baran,
and Andras Hajdu, Senior Member, IEEE
Abstract— Diabetic retinopathy (DR) and especially diabetic
macular edema (DME) are common causes of vision loss
as complications of diabetes. In this work, we consider an
ensemble that organizes a convolutional neural network (CNN)
and traditional hand-crafted features into a single architecture
for retinal image classification. This approach allows the joint
training of a CNN and the fine-tuning of the weights of hand-
crafted features to provide a final prediction. Our solution
is dedicated to the automatic classification of fundus images
according to the severity level of DR and DME. For an objective
evaluation of our approach, we have tested its performance on
the official test datasets of the IEEE International Symposium
on Biomedical Imaging (ISBI) 2018 Challenge 2: Diabetic
Retinopathy Segmentation and Grading Challenge, section B.
Disease Grading: Classification of fundus images according to
the severity level of diabetic retinopathy and diabetic macular
edema. As for our experimental results based on testing on
the Indian Diabetic Retinopathy Image Dataset (IDRiD), the
classification accuracies have been found to be 90.07% for the
5-class DR challenge, and 96.85% for the 3-class DME one.
Index Terms— diabetic retinopathy screening, hand-crafted
features, deep learning, ensemble learning
I. I NTRODUCTION
Diabetic Retinopathy (DR) is an eye disease associated
with long-standing diabetes and the leading cause of blind-
ness in the working-age population of the developed world.
In 2015, an estimated 30.3 million people of all ages (or
9.4% of the population) had diabetes in the U.S. and the
World Health Organization estimates that 422 million people
have the disease worldwide [1]. Around 40% to 45% of US
citizens with diabetes have some stage of DR. Progression to
vision impairment can be slowed down or prevented if DR is
detected in time; however, it can be difficult as the disease
often shows few symptoms until it is too late to provide
efficient treatment.
Clinical experts can identify DR by the presence of lesions
associated with the vascular abnormalities caused by the
disease. While this approach is effective, its resource needs
are high, and there is a lack of expertise and required
equipment in many areas where the rate of diabetes is high.
As the number of individuals with diabetes is growing,
the infrastructure needed to prevent blindness due to DR
will be more and more difficult to access. The need for a
*This work was supported in part by the projects EFOP-3.6.2-16-2017-
00015 and EFOP-3.6.3-VEKOP-16-2017-00002 supported by the European
Union, co-financed by the European Social Fund.
Balazs Harangi, Agnes Baran, Andras Hajdu and Janos Toth are with
the Faculty of Informatics, University of Debrecen, POB 400, H-4002,
Debrecen, Hungary (phone: +36-52-512-900/75121; e-mail: harangi.balazs,
baran.agnes, hajdu.andras, toth.janos@inf.unideb.hu).
978-1-5386-1311-5/19/$31.00 ©2019 IEEE
comprehensive and automated method of DR screening has
long been recognized, and previous efforts have made good
progress using image classification, pattern recognition, and
machine learning. Recently, deep learning is increasingly
applied in medical image analysis, and a number of such
methods have been proposed where CNNs are used for
microaneurysm or exudate detection (e.g., [2], [3]).
In the last year, the organizers of the 2018 ISBI Challenge
on Diabetic Retinopathy - Segmentation and Grading sub-
challenge 2 called for participation in developing efficient
methods to classify RGB fundus images according to the
severity level of DR and diabetic macular edema (DME).
DR is referred as a clinical diagnosis, characterized by the
presence of one or more retinal lesions like microaneurysms
(MA), hemorrhages (HE), hard exudates (EX), and soft
exudates (SE). DME is a complication associated with DR in
which retinal thickening or accumulation of fluid can occur
at any stage of DR. The risk of having DME is classified
into no risk and two probable risks based on the location
of HEs. According to the instructions of the challenge, two
classification tasks had to be addressed. First, the input image
had to be classified according to the five different stages of
DR. Second, it had to be labeled by the three stages of DME
(see Fig. 1 for examples of the different DR/DME stages).
In this paper, we combine the powerful, self-extracted,
CNN-based features with traditional, hand-crafted ones into
a single framework to enhance classification performance.
Namely, we have extracted 68 conventional features (see
Section 2 for details) and modified the CNN AlexNet [4] to
compose a combined system, where the traditional features
are embedded into the CNN via a fully-connected (FC) layer.
In this way, we have created a single network architecture,
which can be trained by back-propagation in the usual
way for neural nets. During the training stage the pre-
trained AlexNet is fine-tuned and the weights of hand-crafted
features are optimized together. Our experimental evaluation
was performed on the publicly available IDRiD database [5]
collated by a retinal specialist in Nanded, Maharashtra, India.
II. M ETHODOLOGY
In this section, we give an overview of the image level and
lesion specific methods used to extract the traditional features
considered in our system. We also describe how the CNN
and conventional features are fused in a single architecture.
2699
Fig. 1. Sample images from the training dataset of the ISBI 2018 challenge: (a) DR0 and DME0; (b) DR1 and DME1, (c) DR2 and DME2; (d) DR3;
(e) DR4.

A. AM-FM Based Image Level Feature Extraction confidence levels, and the number of candidates is counted
at each level in order to obtain six features.
The amplitude-frequency modulation (AM-FM) method
extracts information from an image by decomposing its green 2) Exudate specific features: Features related to EXs are
channel at different scales into AM-FM components [6]. The important in assessing the severity of non-proliferative DR
different scales are obtained using a set of twenty-five band- and the risk of DME. EXs are lipid residues of serous leakage
pass channel filters associated with four frequency scales. from damaged capillaries and appear as bright spots with
After all image features are extracted, k-means clustering is varied shapes in retinal images. The combined output of the
used to cluster this information into 30 groups. As a result, EX detector ensemble is obtained as follows: each hPP, CEi
a feature vector is obtained, which reflects the intensity, pair produces a binary mask that contains EX candidates.
geometry and texture of the structures contained in the image Then, a probability map is generated using the output of the
[7]. different pairs, where the probability of a pixel belonging to
an EX is determined by the ratio of the number of pairs that
B. Lesion Specific Feature Extraction marked the pixel as EX to the total number of pairs.
To extract lesion specific features related to MAs and EXs, For the construction of the ensemble discussed above,
we used two detector ensembles, which consist of a set of we used the results of Nagy et al. [16]. Four PPs (gray-
hpreprocessing method, candidate extractori pairs (hPP, CEi world normalization (GN) [17], illumination equalization
for short) organized into a voting system. Such a hPP, CEi (IE) [10], morphological contrast enhancement (MC) [18],
pair is formed by applying the PP to the retinal image and and vessel removal with inpainting (VR) [11]) and three CEs
the CE to its output. This way, a hPP, CEi pair extracts a (the methods of Sopharak et al. [19], Walter et al. [20], and
set of lesion candidates from the input image, acting like a Welfer et al. [21]) were used. The eight hPP, CEi pairs used
single detector algorithm. in the ensemble are shown in Table I (b).
1) Number of microaneurysms: MAs are the earliest signs
of DR and indicators of its development, therefore their TABLE I
number is a key feature in DR classification. MAs are T HE hPP, CEi PAIRS OF THE ( A ) MA, AND ( B ) EX DETECTOR
swellings of the capillaries and appear as small red dots; ENSEMBLES .
however, their detection is difficult due to their similarity to
vessel fragments. The combined output of the MA detector (a)
PP CE (b)
ensemble is obtained in the following manner: the candidates PP CE
1 NP Lazar et al.
of the result sets of the hPP, CEi pairs are merged, if their 1 GN Sopharak et al.
2 CLAHE Lazar et al.
2 MC Sopharak et al.
Euclidean distances are smaller than a predefined value. To 3 IE Lazar et al.
3 VR Sopharak et al.
each joint candidate of the ensemble a confidence value is 4 WK Lazar et al.
4 IE Walter et al.
assigned showing the ratio of the number of hPP, CEi pairs 5 NP Walter et al.
5 MC Walter et al.
6 CLAHE Walter et al.
suggesting this candidate to the total number of pairs of the 6 VR Walter et al.
7 NP Zhang et al.
ensemble. 7 IE Welfer et al.
8 VR Zhang et al.
8 MC Welfer et al.
To construct the hPP, CEi pairs of the ensemble, we 9 WK Zhang et al.
relied on the results of Antal and Hajdu [8]. Five PPs
(contrast limited adaptive histogram equalization (CLAHE)
[9], illumination equalization (IE) [10], vessel removal with Finally, the result set of the ensemble is thresholded at
inpainting (VR) [11], Walter-Klein contrast enhancement eight different confidence levels, and the following features
(WK) [12], and no preprocessing (NP) for formal reasons) are calculated at each level to obtain a total of 32 features:
and three CEs (the methods of Lazar et al. [13], Walter et ratio of all EX pixels to ROI pixels, number of EXs (8-
al. [14], and Zhang et al. [15]) were used; the pairs of our connected components), ratio of the largest EX (8-connected
MA ensemble are listed in Table I (a). Finally, the result component) to the ROI, and ratio of the average EX size to
candidate set of the ensemble is thresholded at six different the ROI.

2700
 Fig. 2. Visual explanation of the hand-crafted feature extraction and their combination with CNN features.
C. The Fusion of CNN and Hand-Crafted Features
To build our network, we have extended the FC layer
F Cf use originally containing 4096 neurons of AlexNet pre-
ceding the FC layer F Cclass reducing these weights to class
probabilities (5 classes for DR, 3 classes for DME). That
is, we have added a 68-dimensional vector containing the
traditional features to F Cf use to gain 4164 features as it can
be seen in the visual architecture of the proposed ensemble
framework on Fig. 2. Then, the 4164×5 (resp. 4164×3) layer
F Cclass is considered for the DR (resp. DME) classification
task. In this way, both the final weighting F Cclass of the
CNN/traditional features and the 4096 AlexNet features
could be trained by backpropagation. To have a more precise
(i)
formal description, for a given training image x(i) , let y1
(i)
and y2 denote the feature vectors provided by AlexNet and
the traditional extractors, respectively. These two vectors are
combined by F Cclass using the weight matrices A1 and A2
via
(i) (i)
y (i) = AT1 y1 + AT2 y2 , (1)
where A1 has the size 4096×5 (resp. 4096×3), and A2 is of
68×5 (resp. 68×3) for grading DR (resp. DME). As usual in
CNN architectures, F Cclass is followed by a softmax layer
to have the predictive class probabilities yb(i) for the image
x(i) , and by a final classification layer. To train our network,
we have used the mean squared error loss function
M 2
1 X  (i)
L= yb − d(i) ,
2M i=1
where d(i) is the ground truth label for the ith training image
x(i) , while M is the cardinality of the training dataset. The
entries of the weighting matrices A1 and A2 are trainable
parameters of the network; they are initialized with random
values.
III. EXPERIMENTAL RESULTS
For experimental analyses, we have considered the dataset
made available during the ISBI 2018 challenge [22]. The
organizers shared data for both the DR and DME classifica-
tion tasks. The DR task is a 5-class classification problem to
assign grades 0 to 4 according to the severity of the disease
and regarding the 3-class DME task. In the Indian Diabetic
Retinopathy Image Dataset [5], 413 images are available for
training and 103 for testing for both classification problems.
The number of training images is rather low for efficient
deep learning and there are too few images in the test set
as well for comprehensive performance evaluation. Thus,
to increase training data, we have included a large dataset
(22,700 images) made publicly available in a 2015 DR
challenge on Kaggle [23]. This dataset contains ground truth
for the DR task, while the missing DME labels were assigned
by a local ophthalmologist for 11,600 images of this set.
For the measurement of the system accuracy, we used all
the images (516) from the IDRiD dataset. Thus, for training
purposes we have used the Kaggle training dataset, and tested
classification performance on the ISBI 2018 data.
To evaluate our approach, we have considered the common
(2)
error measure accuracy corresponding to a given confidence
2701
level on the receiver operating characteristic curve. However,
it is the simplest way to measure the performance of our
methods regarding the 5- and 3-class classification tasks. An
image is a true case if the predicted label and the ground truth
label match, else it is a false case. To calculate accuracy the
total number of true cases is divided by the total number
of input images. In this way, the classification performances
have been found to be 90.07% for the 5-class DR challenge,
and 96.85% for the 3-class DME one, respectively.
As a comparative analysis between our method and other
approaches in this specific field, we have involved all the
state-of-the-art approaches into the quantitative compari-
son that were presented during the 2018 ISBI Challenge
on Diabetic Retinopathy - Segmentation and Grading sub-
challenge 2. Notice that, in this challenge just a subset of
the IDRiD dataset [5] was considered for the evaluation and
the organizers calculated the average of the 5- and 3-class
classification accuracy for the final scoreboard. According
to this metric, our approach (HarangiM2) finished at the
4th place and outperforms 9 other solutions from the 13
submitted to the challenge as can be seen in Table II.
We note here that some of the algorithms that were ranked
higher in this challenge used information about the number
of images in the different classes of the training set for the
final decision. Such a bias can be an advantage in an artificial
environment like a challenge but it could not be exploited in
a real-world application.
TABLE II
T HE OFFICIAL RESULT OF THE 2018 ISBI C HALLENGE ON D IABETIC
R ETINOPATHY - S EGMENTATION AND G RADING SUB - CHALLENGE 2.
No. Team name Authors Accuarcy
1 Mammoth Junyan Wu et al. 0.9322
2 SDNU Xiaodan Sui et al. 0.8789
3 HarangiM1 Balazs Harangi et al. 0.8741
4 HarangiM2 Balazs Harangi et al. 0.8692
5 AVSASVA Varghese Alex et al. 0.8426
6 VRT Jaemin Son et al. 0.7554
7 LzyUNCC Zhongyu Li et al. 0.5327
8 Py Siddhesh Thakur et al. 0.5206
9 SZU Xuechen Li et al. 0.4964
10 SS K V Sai Sundar et al. 0.4843
11 ZJU-BII-SGEX Xingzheng Lyu et al. 0.3995
12 deepdr Ling Dai et al. 0.3148
13 NTHU-CVLab Chih-Hsuan Liu et al. 0.1719
IV. CONCLUSION
In this work, we have presented an ensemble-based system
dedicated to DR and DME grading. Our ensemble consists
of a convolutional neural network (based on AlexNet) and
several hand-crafted features, traditionally used in fundus
image analysis in the recent decades. The hand-crafted
features are inserted in the CNN architecture to let it be
trained also in consideration of them. As for the performance
of our approach, we have measured 90.07% classification
accuracy for the 5-class DR task, and 96.85% for the 3-class
DME one using the required protocol of the challenge.
2702
R EFERENCES
[1] World Health Organization, Global report on diabetes. World Health
Organization, 2016.
[2] J. Shan and L. Li, “A deep learning method for microaneurysm
detection in fundus images,” in Connected Health: Applications,
Systems and Engineering Technologies (CHASE), 2016 IEEE First
International Conference on. IEEE, 2016, pp. 357–358.
[3] J. H. Tan, H. Fujita, S. Sivaprasad, S. V. Bhandary, A. K. Rao, K. C.
Chua, and U. R. Acharya, “Automated segmentation of exudates,
haemorrhages, microaneurysms using single convolutional neural net-
work,” Information Sciences, vol. 420, pp. 66–76, 2017.
[4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” Commun. ACM, vol. 60,
no. 6, pp. 84–90, 2017.
[5] P. Porwal, S. Pachade, R. Kamble, et al., “Indian Diabetic Retinopathy
Image Dataset (IDRiD),” 2018.
[6] J. P. Havlicek, “AM-FM image models,” Ph.D. dissertation, The
University of Texas at Austin, 1996.
[7] C. Agurto, V. Murray, E. Barriga, S. Murillo, M. Pattichis, H. Davis,
S. Russell, M. Abramoff, and P. Soliz, “Multiscale AM-FM meth-
ods for diabetic retinopathy lesion detection,” IEEE Transactions on
Medical Imaging, vol. 29, no. 2, pp. 502–512, Feb 2010.
[8] B. Antal and A. Hajdu, “Improving microaneurysm detection using
an optimally selected subset of candidate extractors and preprocessing
methods,” Pattern Recognition, vol. 45, no. 1, pp. 264 – 270, 2012.
[9] K. Zuiderveld, “Contrast limited adaptive histogram equalization,” in
Graphics Gems IV, P. S. Heckbert, Ed. Academic Press Professional,
Inc., 1994, pp. 474–485.
[10] A. Youssif, A. Ghalwash, and A. Ghoneim, “Comparative study
of contrast enhancement and illumination equalization methods for
retinal vasculature segmentation,” in Cairo International Biomedical
Engineering Conference, 2006, pp. 1–5.
[11] A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-
based inpainting,” in IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), vol. 2, 2003, pp. 721–728.
[12] T. Walter and J.-C. Klein, “Automatic detection of microaneurysms in
color fundus images of the human retina by means of the bounding box
closing,” in Medical Data Analysis: Third International Symposium
(ISMDA), 2002, pp. 210–220.
[13] I. Lazar and A. Hajdu, “Retinal microaneurysm detection through local
rotating cross-section profile analysis,” IEEE Transactions on Medical
Imaging, vol. 32, no. 2, pp. 400–407, 2013.
[14] T. Walter, P. Massin, A. Erginay, R. Ordonez, C. Jeulin, and J.-
C. Klein, “Automatic detection of microaneurysms in color fundus
images,” Medical Image Analysis, vol. 11, no. 6, pp. 555–566, 2007.
[15] B. Zhang, X. Wu, J. You, Q. Li, and F. Karray, “Detection of
microaneurysms using multi-scale correlation coefficients,” Pattern
Recognition, vol. 43, no. 6, pp. 2237–2248, 2010.
[16] B. Nagy, B. Harangi, B. Antal, and A. Hajdu, “Ensemble-based
exudate detection in color fundus images,” in Symposium on Image
and Signal Processing and Analysis, 2011, pp. 700–703.
[17] G. D. Finlayson, B. Schiele, and J. L. Crowley, “Comprehensive colour
image normalization,” in Computer Vision — ECCV’98, H. Burkhardt
and B. Neumann, Eds. Berlin, Heidelberg: Springer Berlin Heidel-
berg, 1998, pp. 475–490.
[18] P. Soille, Morphological Image Analysis: Principles and Applications.
Springer-Verlag, 2004.
[19] A. Sopharak, B. Uyyanonvara, S. Barman, and T. H. Williamson,
“Automatic detection of diabetic retinopathy exudates from non-dilated
retinal images using mathematical morphology methods,” Comp. Med.
Im. Grap., vol. 32, no. 8, pp. 720–727, 2008.
[20] T. Walter, J.-C. Klein, P. Massin, and A. Erginay, “A contribution of
image processing to the diagnosis of diabetic retinopathy-detection of
exudates in color fundus images of the human retina,” Transactions
on Medical Imaging, vol. 21, no. 10, pp. 1236–1243, 2002.
[21] D. Welfer, J. Scharcanski, and D. Marinho, “A coarse-to-fine strategy
for automatically detecting exudates in color eye fundus images,”
Computerized Medical Imaging and Graphics, vol. 34, no. 3, pp. 228–
235, 2010.
[22] “IEEE International Symposium on Biomedical Imaging 2018 Chal-
lenge 2: Diabetic Retinopathy Segmentation and Grading Challenge,”
https://idrid.grand-challenge.org/, accessed: 2019-02-10.
[23] “Kaggle Diabetic Retinopathy Detection competition,”
https://www.kaggle.com/c/diabetic-retinopathy-detection, accessed:
2019-02-10.