Transfer learning radiomics based on multimodal three-
dimensional MRI/PET imaging for staging Alzheimer’s disease
Abstract
Deep neural networks have been successfully applied
to unsupervised feature learning for single modalities.
Neuroimaging scans acquired from MRI and metabolism
images obtained by FDG-PET provide in-vivo
measurements of structure and function (glucose
metabolism) in a living brain. It is hypothesized that
combining multiple different image modalities providing
complementary information could help improve early
diagnosis of a patient’s mental state: Cognitive Normal
(CN), Mild Cognitive Impairment (MCI), and
Alzheimer’s Disease (AD).
In this work, we propose a novel application of deep
networks to learn features over multiple modalities. We
present a series of tasks for multimodal learning and show
how to train deep neural networks that learn features to
address these tasks. In particular, we demonstrate cross
modality feature learning, where better features for one
modality (e.g., structure) can be learned if multiple
modalities (e.g., structure and function) are present at
feature learning time. Furthermore, we show how to learn
a shared representation between modalities and evaluate it
on a unique task, where the classifier is trained with
structure-only image but tested with function-only data
and vice-versa. Our models are validated on the
Alzheimer’s Disease Neuroimaging Initiative (ADNI)
database [1]. We used FreeSurfer, a brain imaging
software package, to reconstruct the ADNI images and
segment the brain scans into different subcortical
structures (e.g., hippocampus, ventricle, thalamus) [2].
Using these reconstructed 3D images, we were able to
effectively isolate the regions of interest (ROI).
Even though the neuroimaging scans and metabolism
images are providing important information for AD
diagnostics, there are still large portions of the brain that are
unmapped which limits our understanding of the brain. Due to
data collection rates of modern technologies, there is an ever-
increasing volume of brain data collected. These large data
volumes present significant challenges around quality
assurance and validation with current approaches often
requiring manual input.
The aim of this study is to test the efficacy of applying
novel 3D Convolutional Neural Network models to the
problem of staging the AD progression. The network will
analyze important features such as the separation and shapes
of different parts such as the ventricle and hippocampus which
provide additional parameters for learning. The results
reported from hold-out test sets show promising performance
with a comparable classification accuracy. The main limitation
of the 2D slice-level approach is that MRI is 3-dimensional,
whereas the 2D convolutional filters analyze all slices of a
subject independently. Moreover, there are many ways to
select slices that are used as input (as not all of them may be
informative), and slice-level accuracy and subject-level
accuracy are often confused. By comparing the correlation of
multiple modalities, we can determine the dominating feature
in forming the surrogate model. Deploying a sufficiently
trained model in a productionized processing pipeline could be
transformational, reducing the manual intervention required to
locate the information (e.g., hippocampus) needed.
REFERENCES
1. ADNI http://adni.loni.usc.edu/
2. FreeSurfer
https://surfer.nmr.mgh.harvard.edu/fswiki/FreeSurferW
iki