ORIGINAL ARTICLE

Real-time automated diagnosis of precancerous lesions and

early esophageal squamous cell carcinoma using a deep
learning model (with videos)
LinJie Guo, MD,1 Xiao Xiao, PhD,2 ChunCheng Wu, MD,1 Xianhui Zeng, MD,1 Yuhang Zhang, MD,1
Jiang Du, NP,1 Shuai Bai, NP,1 Jia Xie, NP,1 Zhiwei Zhang, MS,2 Yuhong Li, BS,2 Xuedan Wang, BS,2
Onpan Cheung, MD,3 Malay Sharma, MD,4 Jingjia Liu, BS,2 Bing Hu, MD1

Chengdu, Shanghai, China; Rialto, California, USA; Meerut, India

Background and Aims: We developed a system for computer-assisted diagnosis (CAD) for real-time automated
diagnosis of precancerous lesions and early esophageal squamous cell carcinomas (ESCCs) to assist the diagnosis
of esophageal cancer.
Methods: A total of 6473 narrow-band imaging (NBI) images, including precancerous lesions, early ESCCs, and
noncancerous lesions, were used to train the CAD system. We validated the CAD system using both endoscopic
images and video datasets. The receiver operating characteristic curve of the CAD system was generated based on
image datasets. An artificial intelligence probability heat map was generated for each input of endoscopic images.
The yellow color indicated high possibility of cancerous lesion, and the blue color indicated noncancerous lesions
on the probability heat map. When the CAD system detected any precancerous lesion or early ESCCs, the lesion
of interest was masked with color.
Results: The image datasets contained 1480 malignant NBI images from 59 consecutive cancerous cases (sensi-
tivity, 98.04%) and 5191 noncancerous NBI images from 2004 cases (specificity, 95.03%). The area under curve
was 0.989. The video datasets of precancerous lesions or early ESCCs included 27 nonmagnifying videos (per-
frame sensitivity 60.8%, per-lesion sensitivity, 100%) and 20 magnifying videos (per-frame sensitivity 96.1%,
per-lesion sensitivity, 100%). Unaltered full-range normal esophagus videos included 33 videos (per-frame
specificity 99.9%, per-case specificity, 90.9%).
Conclusions: A deep learning model demonstrated high sensitivity and specificity for both endoscopic images
and video datasets. The real-time CAD system has a promising potential in the near future to assist endoscopists
in diagnosing precancerous lesions and ESCCs. (Gastrointest Endosc 2019;-:1-10.)

INTRODUCTION for more than 90% of esophageal cancers in China.

The overall 5-year survival rate is <20%.2 Early
Esophageal cancer is one of the most common ma- diagnosis of precancerous lesions and ESCCs is
lignant cancers worldwide.1 Esophageal squamous cell therefore essential for a favorable prognosis for
carcinoma (ESCC) is the main subtype, accounting patients.

Abbreviations: AI, artificial intelligence; CAD, computer-assisted diag- 0016-5107/$36.00

nosis; ESCC, esophageal squamous cell carcinoma; FN, false negative; https://doi.org/10.1016/j.gie.2019.08.018
FP, false positive; IPCL, intrapapillary capillary loops; NBI, narrow-
Received April 15, 2019. Accepted August 8, 2019.
band imaging; ROC, receiver operating characteristic; TN, true nega-
tive; TP, true positive; WCH, West China Hospital. Current affiliations: Department of Gastroenterology, West China Hospital,
Sichuan University, Chengdu (1); Shanghai Wision AI Co Ltd, Shanghai,
DISCLOSURE: Drs Xiao and Liu are employees and shareholders of
China (2); San Bernardino Gastroenterology Associates Inc and ACE
Shanghai Wision AI Co Ltd. Drs Z. Zhang, Li, and Wang are employees
Endoscopy and Surgery Center, Rialto, California, USA (3); Jaswant Rai
of Shanghai Wision AI Co Ltd. All other authors disclosed no financial
Speciality Hospital, Meerut, India (4).
relationships relevant to this publication.
Reprint requests: Bing Hu, MD, Number 37 Guoxue Road, Chengdu City,
Copyright ª 2019 by the American Society for Gastrointestinal Endoscopy
Sichuan Province, China.

www.giejournal.org Volume -, No. - : 2019 GASTROINTESTINAL ENDOSCOPY 1

Real-time automated diagnosis of esophageal cancer using deep learning
Both nonmagnifying and magnifying narrow-band imag-
ing (NBI) is important for the diagnosis of precancerous le-
sions and ESCCs.3 The brownish area is the main feature of
precancerous lesions and early ESCCs under nonmagnifying
NBI, whereas intrapapillary capillary loops (IPCLs) are key
features under magnifying NBI.4 Unfortunately, it is not
easy to identify these imaging features in ESCC at an early
stage. When NBI was used by an inexperienced
endoscopist, the sensitivity for detecting ESCC was only
53%.5 A recent study on missed esophageal cancer found
that 6.4% of patients had negative endoscopy results
within 3 years before diagnosis.6 Because of the shortage
of trained endoscopists, especially in rural or undeveloped
regions, the ability to detect precancerous lesions and
ESCCs is a significant challenge.
Computer-assisted diagnosis (CAD) using an artificial in-
telligence (AI) system has made remarkable progress in
recent years. Researchers have used the CAD system to
improve the diagnosis of various GI lesions, such as colo-
rectal polyps, gastric ulcers, Helicobacter pylori infections,
and gastric cancer.7-10 The application of CAD in the diag-
nosis of early ESCC has gained much attention recently.
Horie et al11 first used AI to detect esophageal cancer
with a sensitivity of 98% and a positive predictive value
of 40% in 2018. However, in his study, only static images
were tested, and the difference between nonmagnifying
and magnifying settings was not demonstrated. Zhao
et al12 developed a deep learning model based on
magnifying NBI images to investigate automated
classification of IPCLs. However, that study did not use
real-time analysis, and the study findings focused mainly
on the classification of NBI images instead of detection.
A few other studies also used a CAD system to differentiate
cancerous from noncancerous esophageal images using
micro-endoscopy techniques.13 The authors of those
studies did not use white light or NBI images.
Our aim was to develop a CAD system to achieve real-
time automated diagnosis of precancerous lesions and
early ESCCs, in both nonmagnifying and magnifying set-
tings. We hope that our system will improve the diagnosis
of early esophageal malignancies.
METHODS
Training and test datasets
Four institutions were involved in the development of
the CAD system for diagnosing precancerous lesions and
early ESCCs: Endoscopy Center of West China Hospital
(WCH) in Chengdu, China; Jaswant Rai Speciality Hospital,
Meerut, India; San Bernardino Gastroenterology Associ-
ates, Inc and ACE Endoscopy and Surgery Center, Rialto,
California, USA; and Shanghai Wision AI Co Ltd, Shanghai,
China. A total of 2770 NBI images of precancerous lesions
and early ESCCs in 191 cases and 3703 NBI images of
noncancerous lesions in 358 cases, including esophageal
2 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2019
Guo et al
varices in 9 cases, ectopic gastric mucosa in 105 cases,
esophagitis in 64 cases, and normal esophagus in 180
cases, were used to train the CAD system. WCH contrib-
uted 191 cases of precancerous lesions and superficial
ESCCs, 9 cases of esophageal varices, 105 cases of ectopic
gastric mucosa, 64 cases of esophagitis, and 60 cases of
normal esophagus images. San Bernardino Gastroenter-
ology Associates and Jaswant Rai Speciality Hospital
contributed 60 cases of normal esophagus images. All
images were obtained and recorded in 2017. Endoscopic
images were captured using Olympus endoscopes (GIF-
H260Z, EVIS LUCERA CV260 (SL)/CV290 (SL), Olympus
Medical Systems, Tokyo, Japan).
Four datasets were used for validation purposes. For im-
age validation of dataset A, we collected 1480 malignant
NBI images in 59 consecutive cases of precancerous le-
sions or ESCCs from January 2018 to February 2018. All
cases were confirmed histologically. Among these, 32 cases
(35 lesions) were confirmed by endoscopic submucosal
dissection or surgery. The other 27 cases were diagnosed
by biopsy. Suboptimal quality images due to insufficient
air inflation and blurred images were excluded from the
study. Suboptimal image quality was defined if senior en-
doscopists were unable to see enough imaging features
to determine the diagnosis. For image validation of dataset
B, we collected 5191 noncancerous NBI images in 2004
consecutive cases of normal epithelium or benign lesions
of the esophagus. Cases with suboptimal image quality
were excluded. The dataset included normal squamous
epithelium (821 cases from January 2018 to February
2018), normal gastroesophageal junction epithelium (799
cases from January 2018 to February 2018), typical reflux
esophagitis (109 cases from January 2018 to March 2018),
heterotrophic gastric mucosa (154 cases from January
2018 to June 2018), esophageal varices (69 cases from
January 2018 to June 2018), and submucosal tumor (52
cases from January 2018 to June 2018). All diagnoses of
submucosal tumor were confirmed by endoscopic ultraso-
nography. Three of our center’s experienced endoscopists
validated the diagnoses, and the relevant images were
selected. For video validation of dataset C, we selected
27 precancerous lesions and cases of early ESCC that
were recorded from March 2018 to January 2019. All pa-
tients had endoscopic submucosal dissection, and each
diagnosis was confirmed histologically. Each video was
clipped from the time the lesion first appeared in the visual
field until the same lesion disappeared in the visual field
under NBI examination. We clipped nonmagnifying NBI
videos from all 27 cases, and 20 cases were clipped using
magnifying NBI. All nonmagnifying and magnifying videos
were then processed using video-editing software to elim-
inate frames in which our senior endoscopists were unable
to see the detailed imaging features. Finally, for video vali-
dation of dataset D, we selected all 33 cases with normal
esophagus from March 2018 to December 2018 that we re-
corded previously. Three of those cases used magnifying
www.giejournal.org
Guo et al Real-time automated diagnosis of esophageal cancer using deep learning

Figure 1. The architecture and workflow of the deep learning model. An artificial intelligence hot zone image was generated for any input endoscopic
image. The yellow color indicates high possibility of a cancerous lesion, and the blue color indicates a noncancerous lesion. When CAD detects any pre-
cancerous lesion or early ESCC, the lesion of interest is covered with color. CAD, computer-assisted diagnosis; ESCC, esophageal squamous cell carci-
noma; NBI, narrow-band imaging.

NBI. All videos contained unaltered images from the prox-
imal esophagus to the gastroesophageal junction.
Model development
Experienced endoscopists who had at least 5 years of
experience in diagnosing and treating early esophageal
cancer from WCH Endoscopy Center annotated each ma-
lignant endoscopic image. The boundaries of precancerous
lesions and ESCCs in the image were drawn using Adobe
Photoshop software. Those boundaries were used to
represent the actual lesion area within the image.
During the CAD training process, the parameters of the
mathematical function were initially set to random values.
For each annotated image, the location of a lesion
computed by the deep learning function was compared
with the location annotated by the endoscopist during
endoscopy. The parameters of this mathematical function
were then modified slightly to decrease the error on the
same image. The same process was then repeated multiple
times for every image in the training set. The mathematical
function used in this study was based on SegNet architec-
ture (Fig. 1).
SegNet is a deep encoder-decoder architecture for
multi-class pixelwise segmentation. The architecture con-
sists of a sequence of nonlinear processing layers (en-
coders) and a corresponding set of decoders followed by
a pixelwise classifier. Typically, each encoder consists of
one or more convolutional layers with batch normalization
and a ReLU nonlinearity, followed by nonoverlapping max-
pooling and subsampling. The sparse encoding due to the
pooling process is upsampled in the decoder using the
maxpooling indices in the encoding sequence (http://mi.
eng.cam.ac.uk/projects/segnet/). For our models, we
revolved the last 3 convolution layers in the encoder and

www.giejournal.org Volume -, No. - : 2019 GASTROINTESTINAL ENDOSCOPY 3

Real-time automated diagnosis of esophageal cancer using deep learning
the first 3 deconvolution layers in the decoder for better
generalization.
Definition of the analysis
For image evaluation, images of precancerous lesions
and ESCCs were included in dataset A to test the sensitivity
of the model, and noncancerous images were used in
dataset B to test the specificity. For video evaluation of pre-
cancerous lesions and early ESCCs, 3 experienced endo-
scopists from the WCH Endoscopy Center carefully
examined each algorithmically labeled frame in each video.
For dataset C, the sensitivity of the model for each precan-
cerous lesion and early ESCC, as well as each image frame,
was calculated. For specificity evaluation of normal esoph-
agus in dataset D, each labeled frame was counted as false
positive (FP) to provide a per frame specificity. The early
stage of esophageal cancer is defined as mucosal (T1a)
and submucosal (T1b) cancer regardless of lymph node
metastasis.
Statistical analysis
If the detection of labeled algorithm occurred in the
same instance as precancerous lesions and early ESCCs,
the result was considered true positive (TP), and only
one TP was counted for each image, regardless of the num-
ber of times the algorithm detection labels fell on the same
lesion. The absence of algorithmically detected labeling on
precancerous lesions and early ESCCs was counted as false
negative (FN). Per-image sensitivity was therefore defined
as TP divided by the total number of images with precan-
cerous lesions and early ESCCs (sensitivity Z TP/(TP þ
FN)).
If there was no algorithmically detected labeling on an
image without precancerous lesions and early ESCCs, the
image was then counted as true negative (TN). An FP
was defined as any detection label on an area without pre-
cancerous lesions and early ESCCs. Therefore, per-image
specificity was defined as TN divided by the total number
of images without precancerous lesions and early ESCCs
(specificity Z TN/(TN þ FP)).
For video validation of precancerous lesions and early
ESCCs, per frame sensitivity was defined as the number
of TP frames divided by the total number of frames
with precancerous lesions and ESCCs (sensitivity Z TP/
(TP þ FN)). We also measured one additional metric,
per lesion sensitivity, which we defined as the number
of lesions correctly detected by the algorithm in at least
1 frame of each lesion divided by the total number of
lesions.
For video validation of a normal esophagus, per frame
specificity was defined as the number of TN frames divided
by the total number of frames with normal esophagus
(specificity Z TN/(TN þ FP)). Per-case specificity was
defined as the number of cases correctly detected by the
algorithm in all frames of each case divided by the total
number of cases.
4 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2019
Guo et al
All continuous variables are expressed as the mean
within a range. Statistical analyses were conducted using
SPSS, version 16.0 (SPSS Inc, Chicago, Ill, USA).
Ethics
The study was approved by the Ethics Committee of
WCH, Sichuan University (no. ChiECRCT-20180131).
RESULTS
Patient characteristics: imaging features of
esophageal lesions in the validation datasets
We used 4 independent datasets for validation of our
CAD model. Datasets A and B were used for image analysis,
and datasets C and D were used for video analysis. A total
of 2123 cases, including 6671 images and 80 video clips,
were tested. The detailed features of the validation datasets
are listed in Table 1.
In the sensitivity test for datasets A and C, both images
and video clips of precancerous lesions and ESCCs were
used. Most lesions were located in the middle part of the
esophagus. The morphology of the lesions was summa-
rized using the Paris classification.14 Flat and superficial
depressed types were the most common types; 46.8% of
the lesions were classified as mucosal cancers (T1a) in
the image setting and 88.9% were classified as mucosal
cancers (T1a) in the video setting.
In the specificity test, benign diseases were included in
dataset B, whereas only cases with normal esophagus were
used in dataset D. All videos in dataset D were unaltered,
including 30 nonmagnifying videos and 3 magnifying
videos. The mean duration of nonmagnifying and magni-
fying videos in dataset D was 67.2 seconds and 205.8 sec-
onds, respectively. The total number of frames was
50,372 in nonmagnifying videos and 15,433 in magnifying
videos.
When the CAD model detected precancerous lesions or
ESCCs, the area of interest was masked in blue color (Fig. 2).
Imaging diagnosis of the CAD system
The per-image sensitivity of our CAD system for all ma-
lignant images in 59 cases (dataset A) was 98.04%, and the
per-image specificity for 5191 noncancerous NBI images
from 2004 cases (dataset B) was 95.03% (Table 2).
Among malignant images, 32 cases were confirmed as
precancerous lesions or early ESCCs, and the per-image
sensitivity for this group was 97.5%. The per-image
sensitivity for another 27 cases that were diagnosed by
biopsy and contained precancerous lesions, early ESCCs,
or advanced ESCCs was 98.5%. The per-image specificity
of our CAD system for normal epithelium and various
benign diseases varied from 86.1% to 96.8%. Esophagitis
was most frequently misinterpreted by CAD as ESCC
(Table 2). The receiver operating characteristic (ROC)
curve of the CAD system for image analysis was
generated using datasets A and B (Fig. 3). Different
www.giejournal.org
Guo et al Real-time automated diagnosis of esophageal cancer using deep learning

TABLE 1. Patient demographics and clinical characteristics for the validation datasets

Dataset A Dataset B Dataset C Dataset D

Duration of datasets January-February 2018 January-June 2018 March 2018-January 2019 March-December 2018
Data content 59 consecutive cases of Randomly selected 27 randomly selected 33 randomly selected
precancerous lesions/ESCCs 2004 cases of normal precancerous lesions/early cases of normal esophagus
among which 32 cases (total epithelium or benign ESCCs with videos with videos, including 30
35 lesions) were confirmed lesions of the confirmed by post-ESD nonmagnifying videos and
by ESD or surgery; the other esophagus* pathologic examination, 3 magnifying videos
27 cases were diagnosed including 27 nonmagnifying
by biopsy videos and 20 magnifying
videos
Data processing Exclude unclear pictures Exclude unclear Exclude unclear frames Unaltered, full-range
pictures
Patient demographics
Female gender (%) 16.9 54.3 16.9 46.9
Age (years), mean (range) 63.6 (43-78) 46.5 (16-83) 62.5 (46-78) 45.2 (24-71)
Size (mm), mean (range) 34.1 (5-130) NA 23.3 (5-35) NA
Location (upper/middle/low) 10/41/11 NA 3/20/4 NA
Macroscopic type-Paris 5/8/22/20/7 NA 0/5/10/12/0 NA
classification (0-I/IIa/IIb/IIc/II)I
Tumor depth (LGD/HGD/ 3/12/6/8/6/27 NA 3/12/8/1/3/0 NA
LPM/MM/SM/uncertain)
Duration of video (seconds), NA NA 19.7 (5.16-44.56) 79.8 (10.44-348.6)
mean (range)
ESD, Endoscopic submucosal dissection; NA, not applicable; LGD, low-grade dysplasia; HGD, high-grade dysplasia; LPM, laminae propria mucosae; MM, muscularis mucosae;
SM, submucosal layer.
*Normal squamous epithelium (821 cases), normal gastroesophageal junction epithelium (799 cases), typical reflux esophagitis (109 cases), heterotopic gastric mucosa (154
cases), esophageal varices (69 cases), and submucosal tumor (52 cases) were selected from the database of West China Hospital.

probability thresholds for detection were used. The area
under the ROC curve was 0.989.
Video diagnosis of the CAD system
Our video model was capable of processing at least 25
frames per second with a latency period of less than 100
milliseconds in real-time video analysis. Video demon-
stration of CAD for precancerous lesions and early
ESCCs was shown with nonmagnifying (Video 1,
available online at www.giejournal.org) and magnifying
(Videos 2 and 3, available online at www.giejournal.
org) video clips.
The total per-frame sensitivity and specificity of CAD for
datasets C and D were 91.5% and 99.9%, respectively
(Table 2). In dataset C, the per-frame sensitivity of
nonmagnifying video clips was lower than the magnifying
video clips (60.8% versus 96.1%, respectively). Per lesion
sensitivity was 100% in both nonmagnifying and magnifying
video clips. In dataset D, per-case specificity was 90.9%.
For lesions <10 mm, there were 4 cases (49 images) in
image dataset A and 2 cases in video dataset C (2 nonmag-
nifying videos and 1 magnifying video). The per-image or
per-frame sensitivity of the CAD system for these lesions
was 91.8% in the image dataset (Fig. 4), 46.6% in the
nonmagnifying video dataset, and 98.5% in the
magnifying video dataset. Video diagnosis of the CAD
system for lesions <10 mm is demonstrated in Video 4
(available online at www.giejournal.org).
There were 18 cases with irregular cornification (313 im-
ages) in image dataset A and 4 cases in video dataset C (4
nonmagnifying videos and 2 magnifying videos). The per-
image or per-frame sensitivity of the CAD system for these
lesions was 93.6% in the image dataset, 46.9% in the non-
magnifying video dataset, and 85.8% in the magnifying
video dataset. Video diagnosis of the CAD system for le-
sions with irregular cornification is demonstrated in
Video 5 (available online at www.giejournal.org).
DISCUSSION
ESCC is a major global health challenge. Early detection
is essential. However, quality control of endoscopic exam-
ination and the overall shortage of trained endoscopists are
major problems worldwide.15 To enable a CAD system to
serve as “a second observer” in an endoscopic
examination to support nonexperts in the detection of
ESCC and reduce missed diagnoses will require a high-
performing deep learning model.
Early work on deep learning for medical imaging mainly
used past collected images and videos to develop an algo-
rithm retrospectively and then used a small portion of the
remaining collected images as a validation set. Few studies

www.giejournal.org Volume -, No. - : 2019 GASTROINTESTINAL ENDOSCOPY 5

Real-time automated diagnosis of esophageal cancer using deep learning Guo et al

Figure 2. Examples of precancerous lesions and ESCC detection in dataset A. A-D, Nonmagnifying malignant images with different shape and size. E-H,
Magnifying malignant images with different histologic depths (E, low-grade dysplasia; F, high-grade dysplasia; G, laminae propria mucosae; H, muscularis
mucosae). ESCC, esophageal squamous cell carcinoma.

6 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2019 www.giejournal.org

Guo et al Real-time automated diagnosis of esophageal cancer using deep learning

Figure 2. Conituned.

www.giejournal.org Volume -, No. - : 2019 GASTROINTESTINAL ENDOSCOPY 7

Real-time automated diagnosis of esophageal cancer using deep learning Guo et al

TABLE 2. The performance of CAD for validation datasets A to D

Artificial intelligence diagnosis for each narrow-band imaging

image/frame
Pathologic diagnosis Cancerous Noncancerous Sensitivity (%) Specificity (%)

Dataset A All cancerous lesions 1451 29 98.04 –

Precancerous lesions/early ESCCs (32 cases) 670 17 97.5 –
Early and advanced esophageal cancer (27 cases) 781 12 98.5 –
Dataset B All noncancerous lesions 258 4933 – 95.03
Normal esophageal epithelium 133 2999 – 95.8
Normal gastroesophageal junction epithelium 64 1228 – 95.05
Typical reflux esophagitis 36 223 – 86.1
Heterotopic gastric mucosa 9 276 – 96.8
Submucosal tumor 12 115 – 90.6
Esophageal varices 4 92 – 95.8
Dataset C Nonmagnifying videos clips (27 cases) 8069 5208 60.8 –
Magnifying videos clips (20 cases) 86258 3525 96.1 –
Dataset D Nonmagnifying videos (30 cases); magnifying videos (3 cases) 50 65755 99.9
ESCC, Esophageal squamous cell carcinoma.

study, the deep learning model used a training dataset

Figure 3. Receiver operating characteristic curve (ROC) for the image
analysis based on datasets A and B. Different threshold values (0.999,
0.99, 0.97, 0.95, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, and 0.1) were used
to generate the ROC curve. The area under the ROC curve is 0.989.
The threshold value 0.9 had the maximum accuracy (ie, any pixel with
probability larger than 0.9 is classified as being a precancerous lesion or
ESCC).
have used a forthcoming collected dataset for validation.7
We believe that using a validation dataset that was
collected later in time (on consecutive patients) than
when the development dataset was collected can best
simulate data in a prospective clinical trial, and this is a
necessary step toward a prospective clinical trial. In this
collected in 2017, and validation datasets were collected
between January 2018 and January 2019. Previous work
on deep learning for early detection of ESCC was only
validated on still images with limited scale,11 and the
performance of nonmagnifying and magnifying imaging
was not evaluated sufficiently.11,12 In clinical settings, endo-
scopists detect early ESCCs under both nonmagnifying and
magnifying NBI in real-time video streams. In our study,
however, nonmagnifying and magnifying datasets were
validated on a relatively large scale with a combined total
of 175,536 images and video frames, which is more than
27 times the volume of the training dataset with 6473 im-
ages. Our study also indicated that the locations of lesions
computed by the deep learning function were close to the
locations annotated by expert endoscopists during endos-
copy. It was evident that, with the right training data and
training skills, the CAD system and its function seemed
to provide promising potential in computing the precise
locations of esophageal precancerous lesions and ESCCs.
Compared with the detection of polyps during a colo-
noscopy, which is the most successful area of AI in endos-
copy imaging, the detection of ESCC faces 2 major
technical difficulties.7,16 First, individual heterogeneity is
higher among early ESCCs than colorectal polyps. Early
ESCCs can have a much more diversified morphological
appearance, such as elevation, flat, depression, cornifica-
tion, or various vascular patterns on the mucosal surface.17
Second, both nonmagnifying and magnifying images are
needed for accurate diagnosis of precancerous lesions
and early ESCCs. With magnifying NBI images, the visual
field is dramatically narrowed down to capture only small
portions of a lesion. This markedly jeopardizes definable
imaging features, such as lesion boundary and

8 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2019 www.giejournal.org

Guo et al Real-time automated diagnosis of esophageal cancer using deep learning

Figure 4. Examples of lesions <10 mm. A, Early esophageal cancer, PT1a-muscularis mucosae; B, high-grade dysplasia; C, high-grade dysplasia; D, low-
grade dysplasia.

www.giejournal.org Volume -, No. - : 2019 GASTROINTESTINAL ENDOSCOPY 9

Real-time automated diagnosis of esophageal cancer using deep learning
morphological changes. It is therefore not always possible
for an AI system to detect precancerous or early cancerous
lesions with 100% accuracy in this circumstance. In our
study, local features of precancerous lesions and early
ESCCs under NBI were mainly considered in the
development of our deep learning model, such as
background color and the shape of IPCL. The deep learning
model was built using a neural network with a small
receptive field to scrutinize the subtle local features, as well
as with high nonlinearity to identify individual
heterogeneity in precancerous lesions and early ESCCs.4,7,17
The technological characteristics of our deep learning
model explains the per-frame sensitivity for magnifying
videos (96.1%) and nonmagnifying videos (60.8%). As in
nonmagnifying videos, motion artifacts do occur and could
affect and blur the imaging features. The system might
therefore receive insufficient details to trigger a detection
signal. The false negatives in our video analysis concen-
trated on this limitation (blurry image frames, dim lighting,
lesions too far away, and those lesions with irregular corni-
fication, and lesions with limited visibility due to blood).
The false positives were concentrated more in reflux
esophagitis. Additional training images of relevant positive
and negative samples could be used to improve the cur-
rent sensitivity and specificity.
Our study has several limitations. First, the deep
learning model is only engineered to detect precancerous
lesions and early ESCCs under NBI. Lesions under white-
light imaging and advanced esophageal adenocarcinoma
were not addressed in this study. Second, because the
CAD model was trained to detect features of background
color and IPCL, if the lesion in the validation dataset is
totally or largely masked by irregular cornification, so
that features of background color and IPCL are totally un-
seen, then the CAD model might not be able to detect
the lesion appropriately. Third, we excluded suboptimal
quality images in the validation dataset, which may cause
preselection bias. Also, the validation datasets were from
a single center, so more validations should be done with
open databases or datasets from other hospitals. Fourth,
high sensitivity and specificity of the deep learning model
and the real-time capability of the CAD systems do not
directly lead to advances in clinical care, so randomized
prospective controlled trials should be designed to validate
the applicability of this CAD system.
In conclusion, AI offers a possible solution to enhance
quality control and increase the detection rate of precan-
cerous lesions and early ESCCs, especially for junior endo-
scopists. We demonstrated in this study that our deep
learning model can achieve high sensitivity and specificity
in the diagnosis of precancerous lesions and early esopha-
geal cancer both in image and video settings. Future effort
is warranted to build a more accurate and comprehensive
CAD system. Randomized prospective clinical trials should
be conducted to validate the clinical impact of the CAD
system.
10 GASTROINTESTINAL ENDOSCOPY Volume -, No. - : 2019
Guo et al
ACKNOWLEDGMENTS
This research was funded by Sichuan Science and Tech-
nology Department Key R&D projects (grant no.
2019YFS0257) and by Chengdu Technological Innovation
R&D Projects (grant no. 2018-YFYF-00033-GX).
REFERENCES
1. Rustgi AK, El-Serag HB. Esophageal carcinoma. N Engl J Med 2014;371:
2499-509.
2. Chen XX, Zhong Q, Liu Y, et al. Genomic comparison of esophageal
squamous cell carcinoma and its precursor lesions by multi-region
whole-exome sequencing. Nat Commun 2017;8:524.
3. Nagami Y, Tominaga K, Machida H, et al. Usefulness of non-magnifying
narrow-band imaging in screening of early esophageal squamous cell
carcinoma: a prospective comparative study using propensity score
matching. Am J Gastroenterol 2014;109:845-54.
4. Minami H, Isomoto H, Inoue H, et al. Significance of background color-
ation in endoscopic detection of early esophageal squamous cell car-
cinoma. Digestion 2014;89:6-11.
5. Ishihara R, Takeuchi Y, Chatani R, et al. Prospective evaluation of
narrow-band imaging endoscopy for screening of esophageal squa-
mous mucosal high-grade neoplasia in experienced and less experi-
enced endoscopists. Dis Esophagus 2010;23:480-6.
6. Rodriguez de Santiago E, Hernanz N, Marcos-Prieto HM, et al. Rate of
missed oesophageal cancer at routine endoscopy and survival out-
comes: a multicentric cohort study. United European Gastroenterol J
2019;7:189-98.
7. Wang P, Xiao X, Glissen Brown JR, et al. Development and validation of
a deep-learning algorithm for the detection of polyps during colonos-
copy. Nat Biomed Eng 2018;2:741-8.
8. Hirasawa T, Aoyama K, Tanimoto T, et al. Application of artificial intel-
ligence using a convolutional neural network for detecting gastric can-
cer in endoscopic images. Gastric Cancer 2018;21:653-60.
9. Yamada S, Doyama H, Yao K, et al. An efficient diagnostic strategy for
small, depressed early gastric cancer with magnifying narrow-band im-
aging: a post-hoc analysis of a prospective randomized controlled trial.
Gastrointest Endosc 2014;79:55-63.
10. Nakashima H, Kawahira H, Kawachi H, et al. Artificial intelligence diag-
nosis of Helicobacter pylori infection using blue laser imaging-bright
and linked color imaging: a single-center prospective study. Ann Gas-
troenterol 2018;31:462-8.
11. Horie Y, Yoshio T, Aoyama K, et al. Diagnostic outcomes of esophageal
cancer by artificial intelligence using convolutional neural networks.
Gastrointest Endosc 2019;89:25-32.
12. Zhao YY, Xue DX, Wang YL, et al. Computer-assisted diagnosis of early
esophageal squamous cell carcinoma using narrow-band imaging
magnifying endoscopy. Endoscopy 2019;51:333-41.
13. Swager AF, van der Sommen F, Klomp SR, et al. Computer-aided detec-
tion of early Barrett’s neoplasia using volumetric laser endomicro-
scopy. Gastrointest Endosc 2017;86:839-46.
14. Endoscopic Classification Review Group. Update on the paris classifica-
tion of superficial neoplastic lesions in the digestive tract. Endoscopy
2005;37:570-8.
15. Cotton PB. Quality endoscopists and quality endoscopy units. J Interv
Gastroenterol 2011;1:83-7.
16. Byrne MF, Chapados N, Soudan F, et al. Real-time differentiation of
adenomatous and hyperplastic diminutive colorectal polyps during
analysis of unaltered videos of standard colonoscopy using a deep
learning model. Gut 2019;68:94-100.
17. Codipilly DC, Qin Y, Dawsey SM, et al. Screening for esophageal squa-
mous cell carcinoma: recent advances. Gastrointest Endosc 2018;88:
413-26.
www.giejournal.org