Professional Documents
Culture Documents
Abstract
Recently, there has been great interest in developing Artificial Intelligence (AI)
enabled computer-aided diagnostics solutions for the diagnosis of skin cancer.
With the increasing incidence of skin cancers, low awareness among a growing
population, and a lack of adequate clinical expertise and services, there is an im-
mediate need for AI systems to assist clinicians in this domain. A large number
of skin lesion datasets are available publicly, and researchers have developed AI
solutions, particularly deep learning algorithms, to distinguish malignant skin
lesions from benign lesions in different image modalities such as dermoscopic,
clinical, and histopathology images. Despite the various claims of AI systems
achieving higher accuracy than dermatologists in the classification of different
skin lesions, these AI systems are still in the very early stages of clinical appli-
cation in terms of being ready to aid clinicians in the diagnosis of skin cancers.
In this review, we discuss advancements in the digital image-based AI solutions
for the diagnosis of skin cancer, along with some challenges and future opportu-
nities to improve these AI systems to support dermatologists and enhance their
∗ Correspondingauthor
Email address: saeed.hassanpour@dartmouth.edu (Saeed Hassanpour)
1. Introduction
According to the Skin Cancer Foundation, the global incidence of skin cancer
continues to increase [1]. In 2019, it is estimated that 192,310 cases of melanoma
will be diagnosed in the United States [2]. However, the most common forms
of skin cancer are non-melanocytic, such as Basal Cell Carcinoma (BCC) and
Squamous Cell Carcinoma (SCC). Non-melanoma skin cancer is the most com-
monly occurring cancer in men and women, with over 4.3 million cases of BCC
and 1 million cases of SCC diagnosed each year in the United States, although
these numbers are likely to be an underestimate [3]. Early diagnosis of skin
cancer is a cornerstone to improving outcomes and is correlated with 99% over-
all survival (OS). However, once disease progresses beyond the skin, survival is
poor [4, 5].
In current medical practice, dermatologists examine patients by visual in-
spection with the assistance of polarized light magnification via dermoscopy.
Medical diagnosis often depends on the patients history, ethnicity, social habits
and exposure to the sun. Lesions of concern are biopsied in an office setting,
submitted to the laboratory, processed as permanent paraffin sections, and ex-
amined as representative glass slides by a pathologist to render a diagnosis.
AI-enabled computer-aided diagnostics (CAD) solutions are poised to rev-
olutionize medicine and health care, especially in medical imaging. Medical
imaging, including ultrasound, computed tomography (CT), and magnetic res-
onance imaging (MRI), is used extensively in clinical practice. In the derma-
tological realm, dermoscopy or, less frequently, confocal microscopy, allows for
more detailed in vivo visualization of lesioned features and risk stratification
[6, 7, 8, 9, 10]. In various studies, AI algorithms match or exceed clinician
performance for disease detection in medical imaging [11, 12]. Recently, deep
2
learning has provided various end-to-end solutions in the detection of abnormal-
ities such as breast cancer, brain tumors, lung cancer, esophageal cancer, skin
lesions, and foot ulcers across multiple image modalities of medical imaging
[13, 14, 15, 16, 17].
Over the last decade, advances in technology have led to greater accessibility
to advanced imaging techniques such as 3D whole body photoimaging/scanning,
dermoscopy, high-resolution cameras, and whole-slide digital scanners that are
used to collect high-quality skin cancer data from patients across the world
[18, 19]. The International Skin Imaging Collaboration (ISIC) is a driving force
that provides digital datasets of skin lesion images with expert annotations for
automated CAD solutions for the diagnosis of melanoma and other skin cancers.
A wide research interest in AI solutions for skin cancer diagnosis is facilitated by
affordable and highspeed internet, computing power, and secure cloud storage
to manage and share skin cancer datasets. These algorithms can be scalable to
multiple devices, platforms, and operating systems, turning them into modern
medical instruments [20].
The purpose of this review is to provide the reader with an update on the
performance of artificial intelligence algorithms used for the diagnosis of skin
cancer across various modalities of skin lesion datasets, especially in terms of
the comparative studies on the performance of AI algorithms and dermatolo-
gists/ dermatopathologists. We dedicated separate sub-sections to arrange these
studies according to the types of imaging modality used, including clinical pho-
tographs, dermoscopy images, and whole-slide pathology scanning. Specifically,
we seek to discuss the technical challenges in this domain and opportunities to
improve the current AI solutions so that they can be used as a support tool for
clinicians to enhance their efficiency in diagnosing skin cancers.
The major advances in this field came from the work of Esteva et al. [12] who
used a deep learning algorithm on a combined skin dataset of 129,450 clinical
3
and dermoscopic images consisting of 2,032 different skin lesion diseases. They
compared the performance of a deep learning method with 21 board-certified
dermatologists for classification and differentiation of carcinomas versus benign
seborrheic keratoses; and melanomas versus benign nevi. The performance of
AI was demonstrated to be on par with dermatologists performance for skin
cancer classification. Three main types of modalities are used for the skin le-
sion classification and diagnosis in the work described here: clinical images,
dermoscopic images, and histopathology images. In this section, we start with
analysis of publicly available skin lesion datasets, and then we provide different
sub-sections dedicated to the artificial intelligence solution related to each type
of imaging modality.
1. ISIC Archive: The ISIC archive gallery consists of many clinical and der-
moscopic skin lesion datasets across the world such as ISIC Challenges
datasets [21], HAM10000 [22], and BCN20000 [23].
2. Interactive Atlas of Dermoscopy [24]: The Interactive Atlas of Dermoscopy
has 1,000 clinical cases (270 melanomas, 49 seborrheic keratoses), each
with at least two images: dermoscopic, and close-up clinical. It’s available
for research purposes and has a fee of 250 Euros.
3. Dermofit Image Library [25]: The Dermofit Image Library consists of
1,300 high-resolution images with 10 classes of skin lesions. There is a
need for a licensing agreement with a one-off license fee of 75 Euros, and
an academic license is available.
4. PH2 Dataset [26]: The PH2 Dataset has 200 dermoscopic images (40
melanoma and 160 nevi cases). It is freely available after signing a short
online registration form.
5. MED-NODE Dataset [27]: It consists of 170 clinical images (70 melanoma
and 100 nevi cases). This dataset is freely available to download for re-
search.
4
6. Asan Dataset [28, 29]: It is a collection of 17,125 clinical images of 12 types
of skin diseases found in asian people. It is freely available to download
for research.
7. Hellym Dataset [28]: This dataset consists of 125 clinical images of BCC
cases.
8. SD-198 Dataset [30]: The SD-198 dataset is the clinical skin lesion dataset
containing 6, 584 clinical images of 198 skin diseases. This dataset was
captured with digital cameras and mobile phones.
9. SD-260 Dataset [31]: This dataset is a more balanced dataset when com-
pared to the previous SD-198 dataset since it controls the class size dis-
tribution with preservation of 1060 images for each category. It consists
of 20,600 images with 260 skin diseases.
10. Dermnet NZ [32]: Dermnet NZ has one of the largest and most diverse
collections of both clinical, dermoscopic and histology images of various
skin diseases. These images can be used for academic research purposes.
They have additional high-resolution images for purchase.
11. Derm101 [33]: Similar to Dermnet NZ, this resource provides images
within the dermatology atlas. It has different subscription plans to use
the image library.
12. Derm7pt [34]: This dataset has around 2,000 dermoscopic and clinical
images of skin lesions, with a 7-point check-list criteria.
13. The Cancer Genome Atlas [35]: This dataset is one of the largest col-
lections of pathological skin lesion slides with 793 cases. It is publicly
available for anyone in the research community to use.
5
(a) (b) (c) (d)
Figure 1: Illustration of different types of dermoscopic skin lesion where (a) Nevi (b) Melanoma
(c) BCC (d) Actinic Keratosis (e) Benign Keratosis (f) Dermatofibroma (g) Vascular Lesion
(h) SCC [22]
6
lesions and 25 melanoma cases), dermatologists had an average sensitivity
of 86.6% and specificity of 71.3%, while the deep learning method achieved
a sensitivity of 95% and specificity of 63.8%.
3. Brinker et al. [38] compared the performance of 157 board-certified derma-
tologists at 12 German university hospitals with a deep learning method
(ResNet50) for 100 dermoscopic images (MClass-D) consisting of 80 nevi
and 20 melanoma cases. Dermatologists achieved an overall sensitivity of
74.1%, specificity of 60.0% and an AUROC of 0.67 on the dermoscopic
dataset whereas a deep learning method achieved a specificity of 69.2%
and a sensitivity of 84.2%.
4. Tschandl et al. [39] used popular deep learning architectures known as
InceptionV3 and ResNet50 on a combined dataset of 7,895 dermoscopic
and 5,829 close-up lesion images for diagnosis of non-pigmented skin can-
cers. The performance is compared with 95 dermatologists divided into
three groups based on experience. The deep learning algorithms achieved
accuracy on par with human experts and exceeded the human groups with
beginner and intermediate raters.
Clinical images are routinely captured of different skin lesions with mobile
cameras for remote examination and incorporation into patient medical records,
as shown in Fig. 2.. Since clinical images are captured with different cameras
with variable backgrounds, illuminance and color, these images provide different
insights for dermoscopic images.
1. Yang et al. [30] performed clinical skin lesion diagnosis using representa-
tion inspired by the ABCD rule on the SD-198 dataset. They compared
the performance of the proposed methods with deep learning methods and
dermatologists. It achieved a score of 57.62% (accuracy) in comparison
to the best performing deep learning method (ResNet), which achieved
53.35%. When compared to the clinicians, only senior clinicians who have
7
(a) (b) (c) (d)
Figure 2: Illustration of different types of clinical skin lesion where (a) Benign Keratosis (b)
Melanoma (c) BCC (d) SCC [30]
8
(a) (b) (c) (d)
Figure 3: Illustration of different types of histopathology images where (a) Nevi (b) Melanoma
(c) BCC (d) SCC [32]
1. Hekler et al. [42] used a deep learning method (ResNet50) to compare the
performance of pathologists in classifying melanoma and nevi. The deep
learning model was trained on a dataset of 595 histopathology images (300
melanoma and 295 nevi) and tested on 100 images (melanoma/nevi = 1:1).
The total discordance with the histopathologist was 18% for melanoma,
20% for nevi, and 19% for the full set of images.
2. Jiang et al. [43] proposed the use of a deep learning algorithm on smartphone-
captured digital histopathology images (MOI) for the detection of BCC.
They found that the performance of the algorithm on MOI and Whole
Slide Imaging (WSI) is comparable with 0.95 AUC score. They intro-
duced a deep segmentation network for in-depth analysis of the hard cases
to further improve the performance with 0.987 AUC, 0.97 sensitivity, 0.94
specificity score.
9
3. Cruz-Roa et al. [44] used a deep learning architecture to discriminate
between BCC and normal tissue patterns on 1,417 images from 308 Re-
gion of Interests (ROI) of skin histopathology images. They compared the
deep learning method with traditional machine learning with feature de-
scriptors, including the bag of features, canonical and Haar-based wavelet
transform. The deep learning architecture proved superior over the tradi-
tional approaches by achieving 89.4% in F-Measure and 91.4% in balanced
accuracy.
4. Xie et al. [45] introduced a large dataset of 2,241 histopathological images
of 1,321 patients from 2008 to 2018. They used two deep learning archi-
tectures, VGG19 and ResNet50, on the 9.95 million patches generated
on 2,241 histopathological images to test the classification of melanoma
and nevi on different magnification scales. They achieved high accuracy
in distinguishing melanoma from nevi with average F1 (0.89), Sensitivity
(0.92), Specificity (0.94) and AUC (0.98).
10
3.1. Performance of Deep learning and Unbalanced Datasets
Patients medical history, social habits, and clinical meta-data are considered
when making a skin cancer diagnosis. It is very important to know the diagnostic
meta-data, such as patient and family history of skin cancer, age, ethnicity, sex,
general anatomic site, size and structure of the skin lesion, while performing a
visual inspection of a suspected skin lesion with dermoscopy. Hence, only image-
based deep learning algorithms used for the diagnosis of skin cancer falter on key
aspects of patient and clinical information. It is proven in a previous study [37]
that both beginners and skilled dermatologists performance is improved with
the availability of clinical information and that they performed better than deep
learning algorithms. Unfortunately, both patient history and clinical meta-data
are missing in the most publicly available skin lesion datasets.
In the clinical setting, a suspicious lesion is visually inspected with the help
of dermoscopy. The ABCDE rule is considered an important rule for differen-
tiating benign moles (nevi) from melanoma. This includes whether the lesion
is asymmetrical, has irregular borders, displays multiple colors, whether the di-
ameter of the lesion is greater than six millimetres, and if there has been any
evolution or change in the composition of the lesion. Despite the availability of
11
dermoscopic datasets of skin lesions, deep learning algorithms do not work the
same way or look for a pattern similar to the ABCDE rule trusted by clinicians.
It is mainly due to the complexity of pattern recognition for the characteristics
of skin cancers in medical imaging. That is why, despite recent attempts by re-
searchers to demystify the working of deep learning algorithms, such efforts are
still considered as a black-box approach, especially in medical imaging. Since
there are no timeline dermoscopic datasets available publicly, it is not possible
to determine the change of a lesions characteristics according to the evolution
of the ABCDE rule.
A number of skin lesions can mimic skin cancer in both clinical and mi-
croscopic settings, which could result in misdiagnosis. For example, in clinical
and dermoscopic images, seborrheic keratosis can mimic skin cancers including
BCC, SCC, and melanoma. In histopathology images, there are many histo-
logic mimics of BCC such as SCC, benign follicular tumors, basaloid follicular
hamartoma, a tumor of follicular infundibulum, syringoma, and microcystic ad-
nexal carcinoma [49]. Hence, deep learning algorithms, when trained on limited
classes of skin lesions in a dataset, do not reliably distinguish skin cancers from
their known mimics.
12
(a) (b) (c) (d)
Figure 4: Illustration of intra-class dissimilarities in BCC (a) Nodular BCC (b) Superficial
BCC (c) Morphoeic BCC (d) Basosquamous BCC [32]
13
dermatologists would interpret deep learning models outcomes in diagnosing of
skin cancer.
In the current datasets of skin lesions, the dermoscopic images are captured
with high-resolution DSLR cameras and in an optimal environment of lighting
and distance of capture. A deep learning algorithm trained on these high-quality
dermoscopic datasets, achieving a reasonable diagnostic accuracy, could poten-
tially be scaled to smart-phone vision applications. When this model is tested
on multiple smart phone captured images by different cameras in different light-
ing conditions and distances, the same diagnostic accuracy is hard to achieve.
Deep learning algorithms are found to be highly sensitive to which camera de-
vices are used to capture the data, and their performance degrades if a different
type of camera device is used for testing. Patient-provided self-captured skin
images are frequently of low-quality and are not suitable for digital dermatology
[50, 51].
Most of the cases in the current skin lesion datasets belong to fair-skinned
individuals rather than brown or dark-skinned persons. Although the risk of
developing skin cancer is indeed relatively high among the fair-skinned person
population, people with dark skin can also develop skin cancer and are frequently
diagnosed at later stages [52]. Skin cancer represents 4 to 5%, 2 to 4%, 1 to 2%
of all cancers in Hispanics, Asians, and Blacks, respectively [53]. Hence, deep
learning frameworks validated for the diagnosis of skin cancer in fair-skinned
people has a greater risk of misdiagnosing those with darker skin [54]. In a recent
study, Han et al. [28] trained a deep learning algorithm on an Asan training
dataset consisting of skin lesions from Asians. They reported an accuracy of
81% on the Asian testing set, whereas they reported an accuracy of only 56%
on the Dermofit dataset, which consists of skin lesions of Caucasian people.
Therefore, this drop-in accuracy signifies a lack of transferability of the learned
14
features of deep learning algorithms across datasets that contain persons of a
different race, ethnicity, or population.
BCC, SCC and melanoma collectively comprise 98% of all skin cancers.
However, there are other skin cancers, including Merkel cell carcinoma (MCC),
appendageal carcinomas, cutaneous lymphoma, sarcoma, kaposi sarcoma, and
cutaneous secondaries, that are ignored by most algorithms. Beside these rare
skin cancers, there are certain other skin conditions, such as ulcers, skin infec-
tions, neoplasms, and non-infectious granulomas, that could mimic skin lesions.
If deep learning algorithms are trained on datasets that do not have adequate
cases of these rare skin cancers and other mentioned skin conditions, there is a
high risk of misdiagnosis when it is tested on these skin conditions.
In the clinical setting, the diagnosis of skin cancer is made by inspecting the
skin lesion with or without dermoscopy, followed by confirmatory biopsy and
pathological examination. The major issue with the current publicly available
skin lesion datasets is that they lack complete labels related to the diagnosis
performed by a dermatologist. Nevertheless, most of the classification labels
for dermoscopic skin lesion images are determined by pathological examination.
Still, these dermoscopic and clinical skin lesion datasets do not have correspond-
ing pathological classification labels to develop a complete diagnosis pipeline for
AI.
4. Opportunities
15
reported performance evaluations is nowhere near the real-life diagnosis per-
formed by clinicians treating skin cancer. Often, deep learning algorithms are
deemed as opaque, as they only learn from pixel values of imaging datasets and
do not have any domain knowledge or perform logical inferences to establish
the relationship between different types of skin lesions [54]. But, in the future,
deep learning could do very well for the diagnosis of skin cancer with the given
opportunities listed below.
Deep learning networks are often criticised for social biases due to most of
the imaging data belonging to fair-skinned persons. Skin lesion datasets need
to have racial diversity, i.e., they must add equally distributed skin lesion cases
from fair-skinned and dark-skinned people to reduce social or ethnic bias in deep
learning models. The same concern can be extended to age, especially when the
degree of skin aging or surrounding solar damage can influence the dataset and
decision-making.
4.3. Data-Augmentation
16
dataset [22], the skin lesion images were captured at different magnifications or
angles or with different cameras, a process known as natural data augmentation.
Notably, Goyal et al. [57] used deep learning architecture called Faster R-CNN
to develop the algorithm to generate augmented copies similar to the natural
data-augmentation for other skin lesion datasets.
17
4.6. Semantic Explanation of Prediction
To assist clinicians, deep learning algorithms need to provide a semantic
explanation rather than just a confidence score for the prediction of skin lesions.
One possible solution could be for deep learning algorithms using longitudinal
datasets to provide a semantic explanation in terms of the ABCDE rule.
18
4.10. Rigorous Clinical Validation
5. Conclusion
19
Search strategy and selection criteria
References
[1] S. C. Foundation, Skin cancer facts and statistics, Online (Jan. 2017).
URL https://www.skincancer.org/skin-cancer-information/
skin-cancer-facts/general
[2] W. Street, Cancer facts & figures 2019, American Cancer Society: Atlanta,
GA, USA.
20
[7] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, D. Liang,
Accelerating magnetic resonance imaging via deep learning, in: 2016 IEEE
13th International Symposium on Biomedical Imaging (ISBI), IEEE, 2016,
pp. 514–517.
21
[15] J. W. Wei, L. J. Tafe, Y. A. Linnik, L. J. Vaickus, N. Tomita, S. Hassan-
pour, Pathologist-level classification of histologic patterns on resected lung
adenocarcinoma slides with deep neural networks, Scientific reports 9 (1)
(2019) 3358.
22
[23] M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Re-
iter, A. C. Halpern, S. Puig, J. Malvehy, Bcn20000: Dermoscopic lesions
in the wild, arXiv preprint arXiv:1908.02288.
[30] J. Yang, X. Sun, J. Liang, P. L. Rosin, Clinical skin lesion diagnosis using
representations inspired by dermatologist criteria, in: Proceedings of the
23
IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp.
1258–1266.
[33] Derm101.com.
URL https://www.derm101.com/
24
learning outperformed 136 of 157 dermatologists in a head-to-head dermo-
scopic melanoma image classification task, European Journal of Cancer 113
(2019) 47–54.
25
automated basal-cell carcinoma cancer detection, in: International Confer-
ence on Medical Image Computing and Computer-Assisted Intervention,
Springer, 2013, pp. 403–410.
26
miami-dade county, florida, Archives of Dermatology 142 (6) (2006) 704–
708.
[53] H. M. Gloster Jr, K. Neal, Skin cancer in skin of color, Journal of the
American Academy of Dermatology 55 (5) (2006) 741–760.
[56] T.-C. Pham, C.-M. Luong, M. Visani, V.-D. Hoang, Deep cnn and data
augmentation for skin lesion classification, in: Asian Conference on Intelli-
gent Information and Database Systems, Springer, 2018, pp. 573–582.
[61] A. Bissoto, F. Perez, E. Valle, S. Avila, Skin lesion synthesis with genera-
tive adversarial networks, in: OR 2.0 Context-Aware Operating Theaters,
27
Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures,
and Skin Image Analysis, Springer, 2018, pp. 294–302.
28