Deep Learning in Liver Biopsies using Convolutional

Neural Networks

Alexandros Arjmand*, Constantinos T. Angelis*, Alexandros T. Tzallas*, Markos G. Tsipouras†, Evripidis Glavas*,
Roberta Forlano‡, Pinelopi Manousou‡ and Nikolaos Giannakeas*
*
Department of Informatics and Telecommunication, University of Ioannina, Arta, Greece
†
Department of Informatics and Telecommunications Engineering, University of Western Macedonia,
Kozani, Greece,
‡
Liver Unit/ Division of Integrative Systems Medicine and Digestive Disease, Department of
Surgery and Cancer, Imperial College, London, UK

Abstract—Nonalcoholic fatty liver disease (NAFLD) presents a
wide range of pathological conditions, varying from nonalcoholic
steatohepatitis (NASH) to cirrhosis and hepatocellular carcinoma
(HCC). Their prevalence is characterized by increased fat
accumulation and hepatocellular ballooning. They have become a
cause of concern among physicians and engineers, as significant
implications tend to occur regarding their accurate diagnosis and
treatment. Although magnetic resonance, ultrasonography and
other noninvasive methods can reveal the presence of NAFLD,
image quantitative interpretation through histology has become
the gold standard in clinical examinations. The proposed work
introduces a fully automated diagnostic tool, taking into account
the high discrimination capability of histological findings in liver
biopsy images. The developed methodology is based on deep
supervised learning and image analysis techniques, with the
determination of an efficient convolutional neural network (CNN)
architecture, performing eventually a classification accuracy of
95%.
Keywords—Liver Biopsies; Fatty Liver; Hepatocyte
Ballooning; Deep Learning; Convolutional Neural Networks
I. INTRODUCTION
Nonalcoholic fatty liver disease (NAFLD) is the most
common cause of liver ailment worldwide with prevalence
estimates ranging from 25% to 45% in most clinical studies [1].
Its existence is characterized by significant evidence of hepatic
steatosis, as well as other causes of fat accumulation, such as
significant alcohol consumption, long-term use of a steatogenic
medication and monogenic hereditary disorders [2]. These
contribute to the activation of a pro-inflammatory environment
that engenders hepatocellular injury, resulting in a portion of
NAFLD patients developing cirrhosis, but also undergoing liver
transplantation due to complications generated by portal
hypertension, hepatic failure and hepatocellular cancer [3]. On
the other hand, nowadays the physicians’ clinical interest is
focused on nonalcoholic steatohepatitis (NASH), for the reason
that is an evolutionary form of NAFLD [2, 4]. The minimal
histological requirements for the diagnosis of NASH include ≥
5% liver steatosis, lobular inflammation and ballooning of
hepatocytes without any evidence of hepatocellular damage.
Particularly, ballooning degeneration belongs to the class of
chronic liver diseases as it is closely linked to advanced liver
fibrosis.
Even though liver biopsy is currently considered the gold
standard for NAFLD and NASH activity evaluation, it still refers
to an invasive method [5]. Furthermore, since the visual
counting of these findings indicates a difficult and time-
consuming process, modern studies have emphasized the
development of new recognition methods through digital image
processing techniques. Therefore, an automated and accurate
examination would exclude false positive results causing the
misdiagnosis of NAFLD and NASH [6]. It would also contribute
to the improvement of the patient's pathological condition,
through treatment intervention, even if liver transplantation was
required in cases of end-stage decompensated cirrhosis.
The current work presents a novel methodology for the
analysis of multiple hepatic structures from biopsy images, by
building and training two convolutional neural networks with a
common architecture. Deep learning based techniques are
efficient with appropriate parameters particularly in medical
image analysis due to their flexible design and overcoming the
problems caused by hand-crafted features used in traditional
techniques [7-9]. The purpose of the new network topology is to
solve a 4-class prediction problem of different histological
findings, namely: a) ballooned hepatocytes, b) fat droplets, c)
sinusoids and d) veins. The study’s main objectives are the
precise classification of ballooning degeneration and fat
accumulation areas (Fig. 1), as well as the elimination of the
remaining two anatomical structures as findings of liver
diseases. As a final stage, the developed 4-class detection system
can be integrated into a complete methodology for calculating
the fat and ballooning ratio, thereby providing a more
representative state of the patient's clinical condition.

Fig. 1. Fat droplet and balloon cell samples retrieved by a liver biopsy.
This work is partly funded by the project xBalloon, co financed by the
European Union and Greek national funds through the Operational Program for
Research and Innovation Smart Specialisation Strategy (RIS3) of Ipeiros
(Project Code: 5033187)

978-1-7281-1864-2/19/$31.00 ©2019 IEEE 496 TSP 2019

 II. METHODOLOGY
A three-step classification method is developed, which
ultimately leads to the automatic computation of the ballooning
and fat ratio in the entire liver tissue:
• The collection of a sufficient number of isolated training
samples from digitized biopsies, pointing to the 4-class
findings.
• Training two convolutional neural networks carrying
the same architecture, but employing different
optimization algorithms, as well as estimating their
performance in a number of validation images.
• The characterization of unknown anatomical structures,
leading to the isolation and quantification of NAFLD
and NASH prevalence.
A. Histological Features Isolation
All biopsy slides involved in this study come from NAFLD
cases collected at St. Mary Hospital (Imperial College
Healthcare NHS Trust of London, UK). Part of the population
suffers from a high rate of hepatocyte ballooning, while in all
cases large areas of fat accumulation are also found. In our
occasion, all current needle specimens are colored with the gold
standard Hematoxylin and Eosin (H&E) stain to highlight the
regions of interest. Images were scanned and downsampled at
×20 magnification using a Hamamatsu microscope (Hamamatsu
Photonics, Hamamatsu, Japan). This step makes the image data
suitable for digital processing and analysis since they originally
constituted sizes exceeding 10,000 × 10,000 pixels.
An extraction of histological features, as image patches,
from the entire tissue area, is then performed. These patches are
stored separately in four categories implying the number of
individual class objects. According to the above assumption, an
identification label is assigned for each liver class: a) ballooning,
b) fat, c) sinusoid and d) vein. In total, 720 biopsy findings are
provided forming a balanced image dataset (180 samples per
class), which in the next stage can be resized and processed by
the built deep CNNs. The dataset is divided into three subsets,
aiming the training performance and the classification capability
of the two neural networks to be measured: a) 620 training
samples, b) 60 validation samples and c) 40 testing images
unknown to each classifier.
B. Convolutional Neural Network Architecture
In this paper, a CNN topology (Fig. 2) is employed to learn
features from the extracted biopsy findings. In the first phase, a
critical consideration is taken regarding the size of the initial
Fig. 2. The proposed for the experiments CNN arrchitecture.
497
input layer. Since all image patches refer to microscopic samples
with a small magnification factor, the classification model is set
to be trained on 64×64 pixel data, with 3 color channels of RGB
information. Due to the fact that this generates a massive number
of 12,288 connection weights per hidden neuron (in the first
hidden layer), for modeling image data, a series of convolution
operations are utilized for dimensionality reduction and
computational acceleration purposes [10]. The first convolution
layer is set to detect edge and shape features from the raw image
data, by using 64 filters with a 5-by-5 kernel size. For example,
the amount of pure white pixels that characterize either circular
fat cells or irregularly-shaped sinusoids can be initially
determined. Provided that convolution performs a multiplication
of multidimensional array inputs, or else tensors [11], its output
is calculated as:
C
hj i, j = fi × gij i, j (1)
i = 1
th
where hj(x) denotes the j non-activated feature map at a spatial
location (i, j), gij the kernel between hj and the ith input channel
fi, whereas C is the input channels in which the weighted sum
runs [12]. While the suggested number of filters scan the RGB
color channels with a stride of 1, a 2D convolution is executed
by applying the following formula:
fi × gij i, j = fi m, n gij i - m, j - n (2)
m n
It should be noted that in each convolution layer, a zero
padding technique is utilized in order to assign 0 values around
the inputs. This ensures that the kernel will not lead to any data
leakage, but will also maintain the output size equal to the input,
respectively [13]. As mentioned above, all convolution
functions are executed simultaneously. This creates the
probability of a series of unexpected results to occur, especially
during the interference of backpropagation algorithms [11, 14].
For this particular reason, in each layer batch normalization is
set to help in the training process, making it more robust in
coordinated weight updates. During this procedure, a minibatch
H with the size of 64 activations, takes the form of matrix rows
and attempts to normalize the filtered convolution values, before
they are delivered to a nonlinear activation function. To address
the nonlinear nature of the processed data, the Rectified Linear
Unit (ReLU) function: f(x) = max(0, x), is used to solve the
vanishing gradient problem [15]. Now with the aim of moving
the process to the second convolution layer, a 2×2 max pooling
filtering is performed with a stride of 2. This is highly
recommended, as it helps to break up each feature map into
equally sized tiles [14] and to further decrease overfitting by
reducing the spatial size (width and height) of the data
representation [10].
In the second convolution layer, a number of 32 filters with
a 3-by-3 kernel size is set to look for smaller features inside each
liver finding. For instance, the emphasis is currently on
ballooned hepatocytes within a white region and in occurring red
pixels pointing at blood cells of a hepatic vein. Batch
normalization, ReLU activation and max pooling are also
included, while a dropout layer with a 0.5 probability is now
introduced. This process is repeated once more in the third
convolution layer, with the difference that 16 filters with a 3-by-
3 kernel size are now activated. This is achieved with the
purpose of giving even greater emphasis to the textural
differences between the four examined histological findings.
Max pooling is no longer necessary, as the training process is to
make a transition to the upcoming fully connected layers.
At this moment, a dense layer with 4,096 neurons is defined
in order to gather the detected anatomical features from the three
convolution layers, which are further connected to the final
softmax layer. Dense and softmax layer connections act
similarly to a classic artificial neural network, while the softmax
function assigns probability distributions to generate predictions
for the four hepatic classes [16]
C. Optimization Algorithms
So far, several techniques have been incorporated into the
constructed deep model to speed up its training. Now, two
modern optimizers are called to further accelerate the learning
operation and to determine the values that minimize the cost
function. The first optimizer for the methodology solution
comes from applying the Stochastic Gradient Descend with
Momentum (SGDM) algorithm. Specifically, the momentum is
set to accumulate an exponentially decaying average of past
gradients, as it continues to move in their direction [11]. The
second suggested optimizer is the Adaptive moment estimation
(Adam). It belongs to the family of stochastic optimization
algorithms and has gained popularity due to the inclusion of only
first-order gradients and fewer memory requirements [17]. Since
the method owns an adaptive property, it tends to compute
individual adaptive learning rates for different parameters from
the first (mean) and second raw (uncentered variance) moment
estimates of the gradients.
III. RESULTS AND DISCUSSION
In this section and through a series of evaluation steps, the
behavior of the deep supervised models, which are ultimately
intended for objective quantitative measurements of the
examined liver diseases, is monitored. These include the
validation of the training performance, the visualization of the
learned histological features and most importantly, the reliable
identification of unknown testing samples.
A. Training and Validation Results
Having defined the CNN topology, the interest is currently
focused on the training procedure, which is set to run for 30
epochs. An epoch involves a full cycle on the entire training set
498
(620 samples), and this input value proves to be sufficient for the
current amount of data. In each case and at regular intervals
during training, the accuracy of the validation data is calculated.
It is recalled that the validation set is not used to update the
network weights, but to assess how much a model suffers from
either overfitting or underfitting and for an early assessment of
the classification capability to be made.
B. Visualization of Learned Features
The second evaluation step focuses on investigating
ballooned cell and fat droplet features, by observing which areas
of interest are filtered on the first two convolution layers and
how they are finally activated in their corresponding ReLU
functions. A key feature of each convolution layer is that it
converts the color input image into many 2D channels in the
form of a grayscale image (Fig. 3). This refers to a classic and
efficient image processing technique since each pixel involved
in the activation of a channel signifies the same (x, y) spatial
location in the original input image [18]. A channel that is
mostly gray does not activate as strongly as a white pixel. A
similar logic is also followed in each rectified linear unit that
follows each network layer. In ReLUs, each activation is scaled
to a minimum 0 and a maximum 1 value. Consequently, bright
white pixels represent strong positive activations, while pure
black pixels represent strong negatives, respectively. As shown
in Fig. 3, it is found that the channels in the first layer mostly
learn simple features, including edges and cracks contained in
NAFLD findings, while channels in the second layer focus on
more complex features, such as existing hepatocytes in a
ballooning area and the texture of a lipid droplet. It is also
confirmed by the second convolution layer, that dropout leads to
a slight blurring of images, which forces the discriminating
model to learn more important features that are less co-adapted
and lead to better generalization.
C. Testing Results
In order to test the reliability of the developed methodology,
the two trained models are tested to identify 40 unknown liver
structures (10 per class) for the purpose of measuring the
classification accuracy. For the current task, the learning
algorithms by utilizing a function y = f(x), are asked to assign an
input image described by a vector x, to a class identified by a
class label y ∈ {ballooning, fat, sinusoid, vein}. Thus, the
function outputs a probability distribution value for the four
classes within a [0,1] confidence interval. Based on the exported
percentages (Table I), it is clear that the classifiers are more
stable in detecting swollen hepatocytes, as these consist of
multiple changes in the values of their neighbor pixels. They also
successfully achieve a visual separation of circular structures not
always referred to steatotic fat cells, but to hepatic veins, as a
number of red blood cells are included therein.
TABLE I. ACCURACY RESULTS
Classification Results (%)
Deep
Liver Class Performance
Model
ballooning fat sinusoid vein Testing
CNNadam 100 100 70 100 92.5
CNNsgdm 90 100 90 100 95
 Fig. 3. Visualization of histological features learned by the first two convolution layers.
IV. CONCLUSION
In the current study, a deep learning approach for
identifying the presence of various liver biopsy findings is
proposed. The developed methodology focuses on the training
of two convolutional neural networks employing different
optimization algorithms. According to the overall performance,
the supervised models can produce high accuracy results up to
95%, with SGDM being the most efficient optimizer.
Compared to traditional semi-quantitative diagnostic methods,
CNNs provide fully automated activations for detecting diverse
histological features, including edges, shape, pixel intensity and
texture. The trained networks are able to accurately detect the
main differences between ballooned hepatocytes and fat
droplets, two findings responsible for the increasing prevalence
of NAFLD and NASH in recent years.
REFERENCES
[1] M. E. Rinella, “Nonalcoholic fatty liver disease: A systematic review,”
JAMA (Systematic Review), vol. 313, no. 22, pp. 2263–2273, June 2015.
[2] N. Chalasani, Z. Younossi, J. E. Lavine, M. Charlton, K. Cusi, M. Rinella,
S. A. Harrison, E. M. Brunt, and A. J. Sanyal, “The diagnosis and
management of nonalcoholic fatty liver disease: Practice guidance from
the American association for the study of liver diseases,” Hepatology, vol.
67, no. 1, pp. 328–357, January 2018.
[3] P. Angulo, D. E. Kleiner, S. Dam-Larsen, L. A. Adams, E. S. Bjornsson,
P. Charatcharoenwitthaya, P. R. Mills, J. C. Keach, H. D. Lafferty, A.
Stahler, S. Haflidadottir, and F. Bendtsen, “Liver fibrosis, but no other
histologic features, is associated with long-term outcomes of patients with
nonalcoholic fatty liver disease,” Gastroenterology, vol. 149, no. 2, pp.
389–397, August 2015.
[4] Z. M. Younossi, M. Stepanova, N. Rafiq, L. Henry, R. Loomba, H.
Makhlouf, and Z. Goodman, “Nonalcoholic steatofibrosis independently
predicts mortality in nonalcoholic fatty liver disease,” Hepatology
Communications, vol. 1, no. 5, pp. 421–428, June 2017.
[5] N. Fujimori, T. Umemura, T. Kimura, N. Tanaka, A. Sugiura, T.
Yamazaki, S. Joshita, M. Komatsu, Y. Usami, K. Sano, K. Igarashi, A.
Matsumoto, and E. Tanaka, “Serum autotaxin levels are correlated with
499
hepatic fibrosis and ballooning in patients with non-alcoholic fatty liver
disease,” World Journal of Gastroenterology, vol. 24, no. 11, pp. 1239–
1249, March 2018.
[6] E. Goceri, Z. K. Shah, R. Layman, X. Jiang and M. N. Gurcan,
“Quantification of liver fat: A comprehensive review”, Computers in
Biology and Medicine, vol. 71, pp. 174–189, April 2016.
[7] E. Goceri and N. Goceri, “Deep learning in medical image analysis:
Recent advances and future trends”, 11th International Conference on
Computer Graphics, Visualization, Computer Vision and Image
Processing, Lisbon, Portugal, July 21–23, 2017.
[8] E. Goceri and A. Gooya, “On the importance of batch size for deep
learning”, An Istanbul Meeting for World Mathematicians,
Minisymposium on Approximation Theory & Minisymposium on Math
Education, Instanbul, Turkey, July 3–6, 2018.
[9] E. Goceri, “Formulas behind deep learning success”, International
Conference on Applied Analysis and Mathematical Modeling, Instanbul,
Turkey, July 20–24, 2018.
[10] J. Patterson, and A. Gibson, “Deep learning: A practitioner’s approach,”
O’Reilly Media, July 2017.
[11] I. Goodfellow, Y. Bengio, and A. Courville, “Deep learning,” The MIT
Press, November 2016.
[12] V. Andrearczyk, and P. F. Whelan, “Deep learning in texture analysis and
its application to tissue image classification,” Biomedical Image Analysis,
pp. 95–129, January 2017.
[13] A. Geron, “Hands-on machine learning with scikit-learn and tensorflow:
Concepts, tools, and techniques to build intelligent systems,” O’Reilly
Media, April 2017.
[14] N. Buduma, “Fundamentals of deep learning: Designing next-generation
machine intelligence algorithms,” O’Reilly Media, May 2017.
[15] S. Kevin Zhou, H. Greenspan, and D. Shen, “Deep learning for medical
image analysis,” Academic Press, January 2017.
[16] M. V. Hernandez, and V. Gonzalez-Castro, “Medical image
understanding and analysis: 21st annual conference, MIUA 2017,
Edinburgh, UK, July 11–13, 2017, proceedings,” Communications in
Computer and Information Science, Springer, 2017.
[17] D. Kingma, and J. Ba, “Adam: A method for stochastic optimization,”
International Conference on Learning Representations, 2015.
[18] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson,
“Understanding Neural Networks Through Deep Visualization,” Deep
Learning Workshop, 31st International Conference on Machine Learning,
June 2015.