r Human Brain Mapping 38:2875–2896 (2017) r

Your Algorithm Might Think the Hippocampus

Grows in Alzheimer’s Disease: Caveats of
Longitudinal Automated Hippocampal Volumetry

Tejas Sankar,1* Min Tae M. Park,2,3 Tasha Jawa,4 Raihaan Patel,2,8

Nikhil Bhagwat ,2,5,8,9 Aristotle N. Voineskos,5,6 Andres M. Lozano,4
M. Mallar Chakravarty ,2,7,8* and the Alzheimer’s Disease Neuroimaging
Initiative
1
Division of Neurosurgery, Department of Surgery, University of Alberta, Alberta, Canada
2
Cerebral Imaging Centre, Douglas Mental Health University Institute, Montreal, Quebec, Canada
3
Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
4
Division of Neurosurgery, University of Toronto, Toronto, Ontario, Canada
5
Kimel Family Translational Imaging Genetics Research Laboratory, Campbell Family Mental
Health Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada
6
Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada
7
Department of Psychiatry, McGill University, Montreal, Quebec, Canada
8
Department of Biological and Biomedical Engineering, McGill University, Montreal, Quebec,
Canada
9
Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto,
Ontario, Canada

r r

Abstract: Hippocampal atrophy rate—measured using automated techniques applied to structural MRI
scans—is considered a sensitive marker of disease progression in Alzheimer’s disease, frequently used
as an outcome measure in clinical trials. Using publicly accessible data from the Alzheimer’s Disease
Neuroimaging Initiative (ADNI), we examined 1-year hippocampal atrophy rates generated by each of
five automated or semiautomated hippocampal segmentation algorithms in patients with Alzheimer’s
disease, subjects with mild cognitive impairment, or elderly controls. We analyzed MRI data from 398
and 62 subjects available at baseline and at 1 year at MRI field strengths of 1.5 T and 3 T, respectively.
We observed a high rate of hippocampal segmentation failures across all algorithms and diagnostic

Additional Supporting Information may be found in the online
version of this article.
Contract grant sponsor: NIH; Contract grant number: P50
AG05681, P01 AG03991, R01 AG021910, P50 MH071616, U24
RR021382, R01 MH56584, U01 AG024904; Contract grant sponsor:
DOD ADNI (Department of Defense); Contract grant number:
W81XWH-12-2-0012.
Tejas Sankar and Min Tae M. Park contributed equally to the
manuscript.
Data used in preparation of this article were obtained from the
Alzheimer’s Disease Neuroimaging Initiative (ADNI) database
(adni.loni.usc.edu). As such, the investigators within the ADNI
contributed to the design and implementation of ADNI and/or
provided data but did not participate in analysis or writing of
this report. A complete listing of ADNI investigators can be
found at http://adni.loni.usc.edu/wp-content/uploads/how_to_
apply/ADNI_Acknowledgement_List.pdf
*Correspondence to: Tejas Sankar; Division of Neurosurgery,
Department of Surgery, University of Alberta, 2D Surgery, WMC
Health Sciences Centre, 8440-112 Street, Edmonton, AB T6G 2B7,
Canada. E-mail: tsankar@ualberta.ca and M. Mallar Chakravarty;
Douglas Mental Health University Institute, 6875 LaSalle Boulevard,
Montreal, QC H4H 1R3, Canada. E-mail: mallar@cobralab.ca
Received for publication 20 November 2015; Revised 31 January
2017; Accepted 27 February 2017.
DOI: 10.1002/hbm.23559
Published online 15 March 2017 in Wiley Online Library
(wileyonlinelibrary.com).

C 2017 Wiley Periodicals, Inc.

V
 r Sankar et al. r

categories, with only 50.8% of subjects at 1.5 T and 58.1% of subjects at 3 T passing stringent segmenta-
tion quality control. We also found that all algorithms identified several subjects (between 2.94% and
48.68%) across all diagnostic categories showing increases in hippocampal volume over 1 year. For any
given algorithm, hippocampal “growth” could not entirely be explained by excluding patients with
flawed hippocampal segmentations, scan–rescan variability, or MRI field strength. Furthermore, differ-
ent algorithms did not uniformly identify the same subjects as hippocampal “growers,” and showed
very poor concordance in estimates of magnitude of hippocampal volume change over time (intraclass
correlation coefficient 0.319 at 1.5 T and 0.149 at 3 T). This precluded a meaningful analysis of whether
hippocampal “growth” represents a true biological phenomenon. Taken together, our findings suggest
that longitudinal hippocampal volume change should be interpreted with considerable caution as a
biomarker. Hum Brain Mapp 38:2875–2896, 2017. VC 2017 Wiley Periodicals, Inc.

Key words: Alzheimer’s disease; MRI; atrophy; hippocampus; volumetry

r r

been developed for this purpose [Pruessner et al., 2000;

INTRODUCTION
Alzheimer’s Disease (AD) is a neurodegenerative disor-
der characterized by the accumulation of amyloid-beta
and tau proteins in the brain, ultimately leading to
progressive synaptic, neuronal, and axonal loss which
profoundly affect memory and cognitive function [Small &
Duff, 2008]. The neurodegeneration that ultimately leads
to dementia in AD begins in the medial temporal lobe,
before going on to involve neocortical regions [Braak &
Braak, 1991; Braak, de Vos, Jansen, Bratzke, & Braak,
1998], and manifests as atrophy on structural magnetic
resonance imaging (MRI) [Lerch et al., 2008; Sabuncu
et al., 2011, 2012]. Consequently, quantitative MRI volume-
try of the hippocampus has long been proposed as a sensi-
tive and objective biomarker in AD. Smaller hippocampal
volume has consistently been associated with a diagnosis
of AD [Coupe et al., 2011b; Mouiha & Duchesne, 2011;
Sabuncu et al., 2011], while many studies report that the
rate of hippocampal atrophy over time is accelerated in
patients with AD compared to subjects with mild cognitive
impairment (MCI) or healthy controls [Apostolova et al.,
2012; Mouiha & Duchesne, 2011; Sabuncu et al., 2011]. Fur-
thermore, it has been suggested that hippocampal atrophy
may more sensitively and precisely track disease progres-
sion in AD than clinical measures of cognitive function
[Apostolova et al., 2010; Cavedo et al., 2014; Mielke et al.,
2012; McLaren et al., 2012; Sanchez-Benavides et al., 2014].
As a result, a decrease in the expected rate of hippocampal
atrophy in AD patients—or a hippocampal atrophy rate more
closely approximating that observed in normal aging—may
be considered surrogate evidence of a disease-modifying
effect in trials of novel AD therapies. On this basis, hippocam-
pal atrophy has been included as an outcome measure in
several clinical trials of candidate disease-modifying drugs in
AD [Frisoni et al., 2010].
Hippocampal volume and atrophy rate are typically com-
puted from T1-weighted volumetric MRI data. Several man-
ual segmentation protocols—in which the outline of the
hippocampus is traced by an experienced observer—have
Boccardi et al., 2011b; Watson et al., 1997]. To date, it has
been difficult to determine the superiority of any one proto-
col over another [Boccardi et al., 2011a]. A harmonized con-
sensus protocol has recently been developed it is yet to be
widely adopted [Boccardi et al., 2013a, 2013b]. Despite being
considered the neuroanatomical “gold standard” for struc-
tural image analysis, manual segmentation has several pit-
falls. Primarily, manual segmentation is extraordinarily
labor-intensive and time-consuming, and is, as such,
impractical for computing hippocampal volume in large
patient samples. Furthermore, the adoption of any protocol
requires careful assessment of inter-rater and intra-rater reli-
ability to ensure the concordance of hippocampal volumes
between different observers, and the stability of volume esti-
mates from any single observer over time [Pruessner et al.,
2000]. In response to these drawbacks, several semi-
automated and fully automated hippocampal segmentation
algorithms using libraries of manually segmented hippo-
campi as training data have been developed and are now
freely accessible through popular neuroimaging software
packages [Fischl et al., 2002; Patenaude et al., 2011; Pipitone
et al., 2014]. These algorithms permit hippocampal volumes
and atrophy rates to be computed in large imaging datasets
at the click of a mouse button or through the use of a few
simple commands, while minimizing any potential impact
of human bias [Mouiha & Duchesne, 2011; Pipitone et al.,
2014; Sabuncu et al., 2011].
As different segmentation algorithms use slightly differ-
ent a priori anatomical information to identify the bound-
aries of the hippocampus, there are, predictably, slight
differences in absolute hippocampal volume and absolute
rate of hippocampal atrophy generated by each individual
algorithm for a given set of subjects [Barnes et al., 2009;
Mulder et al., 2014]. Such absolute differences can be tolerat-
ed provided that hippocampal volumes and atrophy rates
are at least concordant across algorithms, that is, those sub-
jects whose hippocampi are larger at baseline, or in whom
hippocampal atrophy is occurring more rapidly over time,
ought to be consistently be identified by each individual
algorithm. In the absence of a single gold-standard

r 2876 r
 r Caveats of Longitudinal Automated Hippocampal Volumetry
algorithm, this type of inter-algorithm concordance would
greatly strengthen our confidence in the use of hippocampal
volume as a biomarker to assess the impact of therapies
directed at AD, or perhaps even to track disease progression
at the individual subject level in clinical practice. To date,
there has been a relative paucity of studies addressing the
concordance between hippocampal atrophy rates generated
across various automated segmentation algorithms [Mulder
et al., 2014]. The need for these types of studies is ever more
pressing because of easier access to hippocampal segmenta-
tion algorithms and high-performance computing hardware
by clinicians and scientists who may lack expertise with
quantitative neuroimaging. The need is heightened by the
ongoing development of new segmentation algorithms, and
the development of turnkey commercial services which take
unprocessed MRI data from patients and generate hippo-
campal volumes and atrophy rates in a “black box” manner.
The work in this article is inspired by an observation
made by our group during our study of hippocampal vol-
umes derived from the Alzheimer’s Disease Neuroimaging
Initiative (ADNI, http://www.adni-info.org), a large-scale,
multisite, longitudinal study of the natural history of AD, in
part designed to determine the utility of various imaging
biomarkers in detecting and tracking AD [Mueller et al.,
2005; Holland et al., 2009; Weiner et al., 2012]. While brain
imaging data from ADNI is made freely available to
researchers, the ADNI database also contains a list of longi-
tudinal hippocampal volume estimates, generated by differ-
ent semi-automated and automated methods, for most
subjects. We observed, unexpectedly, that ADNI-reported
hippocampal volumes increased over time in a number of
subjects across the AD, MCI, and healthy control groups.
This observation runs counter to the expected progressive
atrophy of the hippocampus in AD and MCI, and to a lesser
extent, in normal aging [Mouiha and Duchesne, 2011; Sabu-
ncu et al., 2011]. We therefore undertook a study to deter-
mine the incidence of hippocampal volume increases over
time in the ADNI cohort, by each of five different semi-
automated or automated hippocampal segmentation algo-
rithms. Finding a surprisingly large proportion of patients
with hippocampal enlargement, we then examined whether
patients demonstrating volume increases are identified
independently and consistently across these algorithms.
Based on these observations, we examined whether these
volume increases could be explained by well-known sources
of variability in hippocampal segmentation, such as
scan–rescan reliability, or factors influencing MRI acquisi-
tion, such as field strength. Given the use of hippocampal
volume and atrophy rates as biomarkers for disease progres-
sion and transition from NC to MCI to AD [Amaral et al.,
2016; Wolz et al., 2010a; 2010b], the use of atrophy rates may
also provide diagnostic classification accuracy. We further
tested if this was true across the algorithms examined in this
study. Our findings raise some concerns about the reliability
of current segmentation algorithms in assessing hippocam-
pal atrophy rate, and as such serve as a cautionary note to
end users of these algorithms, especially those investigators
r 2877
r
designing clinical trials in AD, or clinicians who may be
tempted to apply their output as a biomarker at the individ-
ual patient level to aid in clinical decision-making.
MATERIALS AND METHODS
Data Acquisition
Data used in this study were obtained from the ADNI data-
base (http://adni.loni.usc.edu/). The ADNI was launched in
2003 by the National Institute on Aging (NIA), the National
Institute of Biomedical Imaging and Bioengineering (NIBIB),
the Food and Drug Administration (FDA), private pharma-
ceutical companies, and nonprofit organizations, as a $60
million, 5-year public–private partnership. The primary goal
of ADNI has been to test whether longitudinal MRI, positron
emission tomography (PET), other biological markers, and
clinical and neuropsychological assessment can be combined
to measure the progression of mild cognitive impairment
(MCI) and early AD. Determination of sensitive and specific
markers of very early AD progression is intended to aid
researchers and clinicians to develop new treatments and
monitor their effectiveness, as well as lessen the time and cost
of clinical trials.
The Principal Investigator of this initiative is Michael W.
Weiner, MD, VA Medical Center and University of Cali-
fornia San Francisco. ADNI is the result of efforts of many
co-investigators from a broad range of academic institu-
tions and private corporations, and subjects have been
recruited from over 50 sites across the U.S. and Canada.
The initial goal of ADNI was to recruit 800 subjects but
ADNI-1 has been followed by ADNI-GO and ADNI-2. To
date, these three protocols have recruited over 1500 adults,
ages 55–90, to participate in the research, consisting of cog-
nitively normal older individuals, people with early or late
MCI, and people with early AD. The follow-up duration
of each group is specified in the protocols for ADNI-1,
ADNI-2, and ADNI-GO. Subjects originally recruited for
ADNI-1 and ADNI-GO had the option to be followed in
ADNI-2. For up-to-date information, see www.adni-info.org.
From the ADNI1:Complete 1Yr 1.5 T standardized dataset
and ADNI1:Complete 1Yr 3 T standardized dataset, we
identified 650 and 142 subjects respectively for whom a base-
line and a 1-year follow-up MRI scan had been completed.
ADNI investigators acquired MRI data using 1.5 T or 3.0 T
scanners (General Electric Healthcare, Philips Medical Sys-
tems or Siemens Medical Solutions) at multiple sites as
described in Jack et al. [2008]. Uniformly preprocessed
images available through the ADNI database were consid-
ered suitable for analyses after quality control. Preprocessing
steps include gradient nonlinearity and intensity inhomoge-
neity correction, and phantom-based distortion correction
[Wyman et al., 2013]. Representative 1.5 T imaging parame-
ters were TR 5 2400 ms, TI 5 1000 ms, TE 5 3.5 ms, flip
angle 5 8, field of view 5 240 3 240 mm, a 192 3 192 3 166
matrix (x, y, and z directions) yielding voxel dimensions of
r
 r Sankar et al.
1.25 mm 3 1.25 mm 3 1.2 mm. Representative 3.0 T imaging
parameters were TR 5 2300 ms, TI 5 850–900 ms, TE 5 3.5
ms, flip angle 5 8–9, field of view 5 256–260 3 240 mm, a
256 3 256 3 160–170 matrix (x, y, and z directions) yielding
voxel dimensions of 1.20 mm 3 1.00 mm 3 1.00 mm.
Hippocampal Segmentation Algorithms
Evaluated
Five different algorithms for segmentation of the hippo-
campus were chosen for evaluation. Amongst these, three
were algorithms from which hippocampal volume data
were already available within the ADNI database, namely:
FreeSurfer [Fischl et al., 2002, 2004], the Quantitative Ana-
tomical Regional Change (QUARC) algorithm from the Uni-
versity of San Diego California (from this point forward
referred to as UCSD) [Holland et al., 2009, 2012; Holland
and Dale, 2011], and Surgical Navigation Technologies
(SNT) [Hsu et al., 2002]. Quality control of segmentations
generated by these algorithms was already available
through the ADNI database and subsequently used in our
evaluations. We also chose two other segmentation algo-
rithms not already available in the ADNI database, namely
FSL FIRST (http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FIRST)
[Patenaude et al., 2011], and a newly derived multi-atlas seg-
mentation algorithm from our group, MAGeT Brain [Chak-
ravarty et al., 2013; Pipitone et al., 2014]. FSL FIRST was
chosen due to its popularity in the field and in other neuro-
imaging studies of the hippocampus, for example the recent
ENIGMA genome-wide association meta-analysis study
[Stein et al., 2012]. MAGeT-Brain was chosen due to our
familiarity with the methodology and our assessment of its
overall quality in comparison to FreeSurfer, FSL FIRST, and
SNT methods [Pipitone et al., 2014; Treadway et al., 2015].
Note that the primary goal of our work was not to simply
compare algorithmic performance, as has been undertaken
in other studies [Mulder et al., 2014; Pipitone et al., 2014],
but moreover to evaluate their performance and concor-
dance in a longitudinal context. These algorithms are
described in more detail below.
FreeSurfer
FreeSurfer is a publicly available image analysis tool
box (https://surfer.nmr.mgh.harvard.edu/) that uses a
fully automated Markov random fields approach to identi-
fy cortical and subcortical structures based on probabilistic
information available from a library of manually segment-
ed images [Fischl et al., 2002, 2004]. Data accessed for this
study were generated using the recently derived minimal-
ly biased processing stream [Reuter et al., 2012] that relies
on within-subject template creation and initializing the
processing for each time point with common information
from the subject template in order to minimize potential
confounds due to over-regularization. Data from ADNI1
(1.5 T) and ADNI2 (3 T) were processed using FreeSurfer
r 2878
r
4.4 and 5.1 respectively, using the longitudinal processing
stream.
UCSD
The UCSD method was developed for accurate quantifi-
cation of both global and local structural changes based on
longitudinal MRI scans [Holland et al., 2009, 2012; Holland
and Dale, 2011]. The methodology relies on the fully affine
matching of longitudinal data, followed by an intensity
alignment stage that corrects for relative B1-induced inten-
sity distortions. This is followed by a nonlinear registration
phase to match longitudinal images using a linear elastici-
ty model which estimates a well-regularized displacement
field to align two images [Holland and Dale, 2011]. The
displaced transformed image will deform cubic voxels into
irregular hexahedra in order to estimate volume changes.
This is done in a hierarchical iterative procedure in order
to capture subtle volumetric differences between longitudi-
nal data from the same individual. FreeSurfer-based seg-
mentations (as above) were used to create the baseline
definitions of the hippocampal segmentations. Final hippo-
campal volumes were estimated using the UCSD proce-
dure as the output of the nonlinear registration phase.
Surgical navigation technologies (SNT)
Semi-automated hippocampal volumetry was carried
out using a commercially available high-dimensional brain
mapping tool (Medtronic Surgical Navigation Technolo-
gies, Louisville, CO) that has previously been validated
and compared to manual tracing of the hippocampus [Hsu
et al., 2002]. Measurements of hippocampal volume was
achieved by manually placing 22 control points as local
landmarks for the hippocampus on the individual brain
MRI data: one landmark at the hippocampal head, one at
the tail, and four per image (i.e., at the superior, inferior,
medial, and lateral boundaries) on five equally spaced sec-
tions perpendicular to the long axis of the hippocampus.
Second, fluid image transformation is used to match the
individual brains to a template brain [Miller et al., 1997].
The voxels corresponding to the hippocampus are then
labeled and counted to obtain volume.
FSL-FIRST
Segmentations were derived using FSL-FIRST, part of
the FSL toolkit [Jenkinson et al., 2012], (http://fsl.fmrib.ox.
ac.uk/fsl/fslwiki/)). FIRST is a model-based segmentation
tool that uses shape and appearance-based models con-
structed from manually segmented images [Patenaude
et al., 2011]. The manual labels are parameterized as sur-
face meshes and modeled as a point distribution model.
Deformable surfaces are used to automatically parameterize
the volumetric labels using constraints to preserve vertex cor-
respondence across the training data. Based on learned mod-
els, FIRST searches through linear combinations of shape
r
 r Caveats of Longitudinal Automated Hippocampal Volumetry
modes of variation for the most probable shape instance giv-
en the observed intensities in a T1-weighted image. FIRST
was implemented using FSL 4.1.9 version. FIRST segmenta-
tions were performed using the run_first_all script according
to the FIRST user guide (http://fsl.fmrib.ox.ac.uk/fsl/
fslwiki/FIRST/UserGuide).
MAGeT Brain
The MAGeT Brain framework [Chakravarty et al., 2013;
Pipitone et al., 2014] is a modified multiatlas framework that
uses a minimal number of well-defined atlases as inputs. In
our implementation we used five high-resolution atlases of the
hippocampus [Winterburn et al., 2013] as input. MAGeT Brain
uses a subset of the data to be segmented to automatically gen-
erate a template library so that the segmentation process can
be bootstrapped using a subset of the data. Template library
images were chosen based on a representative sampling of
subject diagnosis, age, and sex to model neuroanatomical vari-
ability within the ADNI cohort. Seven subjects from each diag-
nostic group (AD, MCI, healthy control) spanning the age
range within each group were chosen. The template library is
generated through automated nonlinear registration between
each of the atlases and each of the subjects in the template
library, yielding five candidate segmentations for each hippo-
campus. The template library now acts like regular input seg-
mentations in a regular multiatlas procedure. Each of the
subjects is then matched to each of the subjects in the template
library to yield 105 candidate segmentations. The final segmen-
tation is created by selecting the most frequently occurring
label at each voxel location [Collins and Pruessner, 2010]. All
images were converted into the MINC file format (http://
www.bic.mni.mcgill.ca/ServicesSoftware/MINC) and all non-
linear registrations were performed using symmetric normaliza-
tion and a cross-correlation objective function as implemented
in the ANTs toolbox [Avants et al., 2008].
Volumetric data generated from the FreeSurfer, UCSD,
and SNT methods were obtained from the ADNI database
(adni.loni.usc.edu) between March 2012 and December
2012. FSL-FIRST and MAGeT Brain were locally processed
for 1.5 T and 3 T data. FreeSurfer (longitudinal, version
5.1.0) was used to process 3 T images as the data were not
available from the ADNI database.
Identification of Subjects Showing
Hippocampal “Growth”
We compared baseline to 12-month hippocampal volumes
generated by each algorithm to identify those subjects show-
ing hippocampal “growth” over time. Simply, we defined
hippocampal growth as occurring in a given subject if that
subject demonstrated a larger hippocampal volume at the
12-month time point compared to baseline. Hippocampal
“growers” were classified as unilateral (i.e., increase over 12
months in volume of either the right or left hippocampus),
bilateral (i.e., increase over 12 months in volume of both
r 2879
r
right and left hippocampi), or average (i.e., increase over 12
months in mean volume of both hippocampi).
Quality Control
Quality control (QC) assessments were performed to iden-
tify and eliminate subjects with hippocampal segmentation
failures which could impact our hippocampal growth analy-
sis. Briefly, QC information for segmentations from the Free-
Surfer and UCSD algorithms was obtained directly from the
ADNI database, while QC was assessed for FSL and MAGeT
using an in-house method. QC could not be assessed for the
SNT algorithm. A detailed description of QC methods follows
below.
FreeSurfer
QC information for individual hippocampal segmentations
was obtained from the ADNI database. According to ADNI,
QC assessment was carried out as outlined in the ADNI
protocols (http://adni.bitbucket.org/docs/UCSFFRESFR/
UCSFFreeSurferMethodsSummary.pdf). QC was assessed in
both regional and global registration and segmentation accu-
racy. For the purposes of this study, we focused on the
ADNI-reported “Overall QC” metric, which is the most rigor-
ous QC measure, meant to reflect the quality of cortical and
subcortical segmentation across the entire brain volume.
Specifically, we considered a given hippocampal segmenta-
tion in a given subject as acceptable only if that subject
achieved an Overall QC metric of “PASS.”
UCSD
QC information in the form of a simple two-point scale
was obtained from the ADNI database, an ADNI-reported
rating of “0” denoted segmentation failure, while a rating
of “1” indicated a passing segmentation.
SNT
Formal QC was not reported in ADNI, and was not per-
formed considering that the SNT algorithm is semi-
automated, depending in part on the selection of anatomi-
cal landmarks on the hippocampus by an expert observer.
As such, it is thought to be resistant to gross segmentation
error, and is frequently considered a “gold-standard” ref-
erence [Leung et al., 2010; Wolz et al., 2010a].
FSL-FIRST, MAGeT Brain
QC was performed by one of the authors (MTMP). Fifteen
representative slices encompassing left and right hippocam-
pal segmentations were used to evaluate QC. Specifically, if
either hippocampus was underestimated or overestimated
by at least 10 voxels in three or more slices then the segmen-
tation did not pass. The MAGeT Brain segmentations used
here have previously been analyzed in a validation study of
r
 r Sankar et al.
this algorithm, and have been found to show good concor-
dance with manual segmentations derived using the Pruess-
ner protocol [Pruessner et al., 2000] as well as SNT-
generated volumes [Pipitone et al., 2014].
QC assessment as described above was used to generate
three subject groups based on increasingly stringent QC
levels:
1. First-pass QC: Subjects passing first-pass QC were
simply those with complete volumetric information
for all algorithms at both time points. First-pass QC did
not involve any specific assessment of segmentation
quality.
2. Methodwise QC: Subjects passing methodwise QC
were those who, for any given algorithm, passed QC by
that particular algorithm at both time points. Obviously,
this resulted in different numbers of subjects surviving
QC for each algorithm.
3. Across-method QC: Among subjects in #1, only those
passing QC by all algorithms at both time points.
Evaluation of Scan–Rescan Reliability
Scan–rescan reliability can be an important source of error
influencing measurement accuracy of longitudinal volumet-
ric changes in brain structure, and has been assessed in stud-
ies which report the output of automatic segmentation
algorithms applied to multiple MRI scans of the same
subject at short intervals [Morey et al., 2010]. Resulting
segmentations and volumetric differences between repeat
scans are then typically used to quantify error limits within
the segmentation algorithm.
We addressed scan–rescan reliability in two ways.
First, we determined literature-reported mean scan–rescan
reliability data for each algorithm:
FreeSurfer, FSL-FIRST, and UCSD
Scan–rescan data were estimated from a previous study
that compared FreeSurfer and FSL segmentations [Morey
et al., 2010]. Data were not available specifically for the
UCSD method, though we expected higher sensitivity in
detecting subtle longitudinal changes for the UCSD
method compared to FreeSurfer [Holland & Dale, 2011].
We therefore used the same threshold for FreeSurfer and
UCSD as a conservative measure of reliability.
Absolute volumetric differences between raters were
quantified from a previous study comparing SNT with
manual segmentation [Hsu et al., 2002].
MAGeT Brain
Images from five subjects scanned within a week locally
were processed with the MAGeT Brain pipeline, and absolute
volumetric differences between time points were quantified.
r 2880
r
Mean scan–rescan reliability data (expressed as a percent-
age in Supporting Information, Table I) for each algorithm
were then used to define a conservative threshold for hippo-
campal growth: “candidate” growers were those subjects in
ADNI who, for a given segmentation algorithm, demonstrat-
ed an increase in hippocampal volume between baseline and
1-year greater than mean scan–rescan reliability error for that
algorithm.
Derived Scan-Rescan Reliability Estimates
Second, we used data obtained from the Open Access
Series of Imaging Studies (OASIS; http://www.oasis-
brains.org/) from the Consortium for Reliability and
Reproducibility (CoRR; website: http://fcon_1000.projects.
nitrc.org/indi/CoRR/html/) [Zuo et al., 2014]. OASIS (1.5
T) and CoRR (3 T) datasets were used to independently
assess the scan–rescan reliability of the FSL, FreeSurfer and
MAGeT Brain segmentation algorithms, as we had ready
access to these algorithms in our lab. A set of 20 (12 female/
8 male, ages 19–34, all right handed) subjects from the
OASIS dataset who underwent a repeat scan within 90 days
were selected, yielding a set of 40 total images. A set of 84
(46 female/38 male, ages 18–62, handedness not available)
subjects from the CoRR dataset who underwent a repeat
scan within 24 days were selected, yielding a set of 168
images. Specifically, images from the HNU_1, IPCAS_1 and
XHCUMS sites were selected from the CoRR dataset.
OASIS and CoRR data were processed independently with
FSL, Freesurfer and MAGeT Brain segmentation algorithms
as described above for the ADNI dataset. FSL First (version
5.0.6) was run using the run_first_all command in accor-
dance with http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FIRST/
UserGuide. The FreeSurfer longitudinal pipeline (version 4.4
for 1.5 T, version 5.1 for 3 T) was run for each scan–rescan
pair according to http://freesurfer.net/fswiki/Longitudinal-
Processing. MAGeT Brain segmentation was run in accor-
dance with https://github.com/CobraLab/MAGeTbrain.
For 1.5 T data, 19 of the 20 baseline images were selected as
templates. For 3 T data, 21 of the 84 baseline images were
selected in a demographically representative manner.
The resulting scan–rescan CoRR-OASIS reliability distri-
butions for each algorithm were used as null distributions
against which ADNI 1-year atrophy rate distributions
were compared. Subjects from ADNI with atrophy rate
outside the right-tail 90th percentile of the scan–rescan dis-
tribution were considered as an alternate definition for
identifying candidate “growers.” We chose a more liberal
90th percentile threshold based on the assumption that the
scan–rescan distributions we computed likely undersam-
ple the true population variance in scan–rescan values. By
generating a null distribution of test-retest reliability error
we considered that single individuals that fell to the edges
of this distribution would be the most likely candidates to
demonstrate plausible biological growth over the course of
1 year. Moreover, if these individuals were most likely to
exhibit hippocampal growth over 1 year, we hypothesized
r
 r Caveats of Longitudinal Automated Hippocampal Volumetry r

TABLE I. Selected demographic and clinical features of subjects by diagnostic category and field strength

AD MCI Normal P value

1.5 T
N 90 179 129 -
Mean age (range) 74.7 (56–89) 74.9 (55–89) 76.2 (62–90) 0.16
#Females 41 60 60 0.050
MMSE (SD) 23.4 (1.90) 27.1 (1.75) 29.1 (1.03) <0.001
3T
N 15 24 23 -
Mean age (range) 72.1 (57–88) 74.2 (56–88) 75.6 (70–85) 0.35
#Females 9 9 15 0.39
MMSE (SD) 23.1 (2.12) 26.92 (2.00) 29.43 (0.73) <0.001

P values are reported for one-way ANOVA for age and MMSE, and for chi-square test (1.5 T) or Fisher’s exact test (3 T) for proportion
of females. MMSE was significantly different between groups.

that this phenomenon should be captured through the dif-
ferent algorithms tested in this manuscript.
In this particular case, it may be more beneficial to identify
high rates of atrophy using a whole-brain or temporal lobe
measures (such as a medial temporal atrophy score). Howev-
er, at the present time, there are limited measures for lateral
ventricle and whole-brain volumes available via ADNI. Con-
sequently, deriving such scores would limit us and potential-
ly bias downstream analysis to a specific method.
Statistical Analysis
General linear models were used to compare baseline
hippocampal volume and hippocampal atrophy rate
between segmentation algorithms and diagnostic groups,
with age and sex used as co-variates. We verified the
normality of the atrophy rate distributions using a Kolmo-
gorov–Smirnov test for normality as these rates are
derived using the ratio of two normally distributed varia-
bles. The v2 test (or Fisher’s exact test for small n) was used
to assess differences in the proportion of QC failures
between algorithms, diagnostic categories, and timepoints,
as well as differences in the proportion of candidate hippo-
campal “growers” by each method. Correction for multiple
comparisons was performed with the Marascuillo procedure
for multiple proportions. To determine if any subset of individ-
uals (i.e., AD with smallest hippocampi) were driving QC fail-
ures, we also split each diagnostic group into quartiles based
on total hippocampal volume. For each method, we deter-
mined if the group representation in the QC failures was dif-
ferent to the QC passes. To assess concordance between
various algorithms in baseline hippocampal volume and hip-
pocampal atrophy rate, the intraclass correlation coefficient
(ICC) was used in a two-way, mixed-model for a single mea-
sure using a consistency agreement definition.
Diagnostic Classification Based on Atrophy Rates
We performed pairwise classification between three
diagnostic categories (AD, MCI, and NC) using
hippocampal atrophy rates derived from the five different
segmentation pipelines described above. To have a consis-
tent set of subjects we used hippocampal volumes that
survived the most stringent level of quality control. We
trained three different machine-learning models: logistic
regression with Lasso, support vector machine, and ran-
dom forest (RF). We only used total bilateral hippocampal
atrophy estimated at 1 year as input.
The model performance was evaluated using 10-fold nested
cross-validation procedure for each input choice. Folds were
stratified according to the proportion of each diagnostic cate-
gory. Data were preprocessed to center at mean and feature-
wise scaled to have unit-variance. (http://scikit-learn.org/
stable/modules/generated/sklearn.preprocessing.scale.html)
Briefly, cross-validation was performed as follows. First, we
divided the input data into 10 subsets. Then each model was
trained using 9 of these 10 subsets and performance was mea-
sured on the held-out test subset. The process was repeated
10 times. Hyperparameters of the model (number of trees,
Lasso penalty, etc.) were determined through a grid search
using an inner cross validation loop created by further divid-
ing the training subset in each round. All models were imple-
mented using scikit-learn toolbox (http://scikit-learn. org/
stable/index.html). The binary classification performance was
measured using receiver operating characteristic (ROC)
curves and area under the curve (AUC) values.
RESULTS
Demographic and Clinical Data
Demographic and clinical features of subjects in all three
diagnostic categories are listed in Table I.
Baseline Hippocampal Volume and
Hippocampal Atrophy Rates are Consistent
With Literature-Reported Data
Figure 1 shows the mean baseline hippocampal volume
and mean hippocampal atrophy rates categorized by

r 2881 r
 r Sankar et al. r

Figure 1.
Hippocampal volumetric data at 1.5 T. (A) Mean left, right, and average baseline hippocampal vol-
ume by segmentation algorithm. (B) Mean left, right, and average 12-month hippocampal atrophy
rate by segmentation algorithm.

segmentation algorithm at 1.5 T (3 T data are shown in
Supporting Information, Fig. 1) at a first-pass level of
QC. We observed that atrophy rates across methods and
field strengths were normally distributed with the excep-
tion of UCSD at 1.5 T (D 5 0.11, P 5 0.01). As a result, we
chose to continue to use the analyses detailed in the Sta-
tistical Methods section of the Methods (Supporting
Information, Table II). Mean 12-month hippocampal atro-
phy rates for subjects with AD ranged from 1.6% to
6.0%, well within the expected range of literature-
reported values [Barnes et al., 2009]. There was no
significant difference in hippocampal atrophy rate across
algorithms.
Automated Hippocampal Segmentation
Algorithms Result in a Sizeable Number of
Hippocampal Segmentation Failures
We found evidence of hippocampal segmentation fail-
ures by every fully automated algorithm (FreeSurfer,
UCSD, FSL, and MAGeT) in every diagnostic category

r 2882 r
 r Caveats of Longitudinal Automated Hippocampal Volumetry r

Figure 2.
Graphical summary of the proportion of hippocampal segmentation failures at 1.5 T by algorithm,
diagnostic category, and time point: (A) baseline and (B) 1-year follow-up.

(i.e., AD, MCI, or NC). Segmentation failures were
observed regardless of scan field strength: the proportion
of segmentation failures for each algorithm and diagnostic
category at 1.5 T is summarized by Figure 2, while the
proportion of failures at 3 T is shown in Supporting Infor-
mation, Figure 2 (raw data on segmentation failures is
found in Supporting Information, Tables III and IV). The
substantial number of segmentation failures, naturally,
caused several subjects to fail more stringent QC assess-
ment. At 1.5 T, across all diagnostic categories, only
approximately half (202 out of 398, 50.8%) of those subjects
with scans available at baseline and 12 months passed
across-method QC (i.e., had acceptable hippocampal seg-
mentations at both time points by all algorithms). Overall,
the proportion of AD patients passing across-method QC
was significantly lower than that of the NC subjects (37.8%
vs 58.9%, v2 5 9.95, P 5 0.007), but not significantly lower
than that of MCI subjects (51.4%). At 3 T, 58.1% (36 out of
62) of all subjects passed across-method QC overall; as at
1.5 T, the proportion of AD patients passing across-
method QC was significantly lower than that of the NC
subjects (40.0% vs 78.2%, v2 5 6.26, P 5 0.04) but not that
of the MCI patients (50.0%).
We considered the possibility that segmentation algo-
rithms might be less accurate when applied to MRI scans
of increasingly atrophic brains. If this were true, scans at
12 months would be more likely to be classified as QC fail-
ures than those at baseline, as atrophy would be expected
to progress over time. To test this hypothesis, we
compared the proportion of QC failures by each algorithm
at baseline and at 12 months separately. At 1.5 T, the pro-
portion of subjects with a failed hippocampal segmenta-
tion was no different at 12 months compared to baseline
for all algorithms (P > 0.05 by v2 test) except UCSD (v2 5
123.31, P < 0.001). Similarly, at 3 T, only the UCSD algo-
rithm had a lower proportion of QC failures at baseline
(P < 0.001, Fisher’s exact test). No specific diagnostic cate-
gory was seen to have an increased proportion of failures
in any method at 1.5 T or 3 T (Supporting Information,
Table V). Interestingly—and uniquely—there were no seg-
mentation failures at baseline by the UCSD algorithm, in
any clinical group or at either field strength.
Automated Hippocampal Segmentation
Algorithms Demonstrate That Hippocampal
“Growth” Occurs in a Substantial Proportion
of Subjects
At 1.5 T, at the first-pass QC level, a substantial propor-
tion of subjects in each of the AD, MCI, and NC groups
demonstrated increases in hippocampal volume over 12
months (i.e., were candidate “growers”) across all algo-
rithms (Table II and Fig. 3). The percentage of candidate
“growers” for each method, diagnosis, and QC level at 3 T
is listed in Supporting Information, Tables VI and VII.
Even among those subjects who passed stringent, across-
method QC, hippocampal “growth” was still observed in

r 2883 r
 TABLE II. Percentage of subjects at 1.5 T showing hippocampal “growth” for all algorithms at all QC levels

Diagnosis FS UCSD SNT FSL MAGeT

First-pass QC AD Left 13.3 6.7 14.4 25.6 22.2

Right 12.2 6.7 11.1 24.4 25.6
Bilateral 5.6 3.3 4.4 11.1 8.9
Average 13.3 5.6 6.7 21.1 21.1
MCI Left 22.4 17.9 31.3 22.9 30.2
Right 22.4 20.1 22.9 24.6 26.3
Bilateral 10.1 8.9 12.3 11.2 12.9
Average 16.8 14.5 25.1 18.4 24.0
Normal Left 37.2 17.8 42.6 38.0 29.5
Right 33.3 20.2 41.9 40.3 37.2
Bilateral 15.5 11.6 29.5 19.4 15.5
Average 36.4 19.4 43.4 38.0 31.8
Methodwise QC AD Left 11.5 4 - 21.0 23.6
Right 9.6 5.3 - 15.8 23.6
Bilateral 5.8 1.3 - 5.3 9.7
Average 11.5 4 - 14.5 22.2
MCI Left 21.6 14.3 - 22.0 29.8
Right 21.6 16.3 - 23.9 26.2
Bilateral 9.6 5.4 - 10.7 12.5
Average 15.2 9.5 - 17.0 23.8
Normal Left 38.0 16.4 - 37.5 28.9
Right 35.0 18.2 - 40.0 38.8
Bilateral 15.0 10.0 - 17.5 16.5
Average 37.0 17.3 - 37.5 33.1
Across-method QC AD Left 8.8 2.9 14.7 23.5 20.6
Right 8.8 5.9 11.8 23.5 29.4
Bilateral 2.9 0 2.9 5.9 5.9
Average 8.8 3.0 2.9 17.7 23.5
MCI Left 17.4 15.2 38.0 20.7 29.4
Right 19.6 18.5 19.6 19.6 26.1
Bilateral 8.7 5.4 14.1 7.6 14.1
Average 15.2 7.6 27.2 15.2 20.7
Normal Left 39.5 18.4 43.4 35.5 38.2
Right 30.3 18.4 46.0 35.5 39.5
Bilateral 15.8 10.5 34.2 11.8 19.5
Average 38.2 18.4 48.7 35.5 38.2

Note that methodwise QC could not be performed for SNT; see text for details.

Figure 3.
Venn diagrams demonstrating overlap of subjects identified as hippocampal “growers” at 1.5 T
across all 5 segmentation algorithms, divided by (a) first pass QC and (b) across-method QC.
Within (a) and (b), clockwise from upper left, individual Venn diagrams represent overlap for left,
right, bilateral, and average hippocampal growth respectively.
 r Caveats of Longitudinal Automated Hippocampal Volumetry r

some subjects in every diagnostic group. In AD patients, TABLE III. Percentage of subjects (number of
the proportion of candidate “growers” in average hippo- “growers”/total # of subjects within DX group) at 1.5 T
campal volume ranged from 2.94% (UCSD algorithm) to showing average hippocampal growth after thresholding
23.53% (MAGeT algorithm). In MCI patients, the propor- for scan–rescan reliability, divided by segmentation
tion of mean hippocampal candidate “growers” varied algorithm
from 7.61% (UCSD) to 27.17% (SNT algorithm). In NC,
FS UCSD SNT FSL MAGeT
between 18.42% (UCSD) and nearly half (48.68%, SNT) of
subjects were mean hippocampal candidate “growers,”
(A) AD 4.4 0 2.2 10.0 6.7
and the SNT, FSL, Freesurfer, and MAGeT algorithms all MCI 5.0 2.2 11.2 6.2 11.2
found that >35% of NC subjects demonstrated growth in Normal 7.8 1.6 21.7 7.8 7.8
average hippocampal volume. Overall, the proportion of (B) AD 1.9 0 2.2 1.3 4.2
average hippocampal candidate “growers” was different MCI 4.0 1.4 11.2 5.0 10.1
between algorithms (omnibus v2 5 26.78, P < 0.0001). Post- Normal 7.0 0.9 21.7 5.0 8.3
hoc testing showed that the proportion of average hippo- (C) AD 2.9 0 0 0 2.9
campal candidate “growers” was lower for the UCSD meth- MCI 2.2 1.1 13.0 4.4 7.6
od compared to any other algorithm (P < 0.035 for all Normal 6.6 0 22.4 2.6 9.2
(D) AD 2.9 N/A N/A 0 0
pairwise comparisons). Owing to smaller sample sizes, the
MCI 0 N/A N/A 3.3 1.1
results at 3 T were more variable (Supporting Information, Normal 5.3 N/A N/A 2.6 1.3
Fig. 3 and Tables VI and VII). Growth rates as a percentage
at 1.5 T are given in Supporting Information, Table VIII. (A) First pass QC, (B) Methodwise QC, (C) Across-method QC,
(D) Across-method QC and using 90th percentile thresholds gener-
ated based on CoRR-OASIS scan–rescan distributions. N/A 5
Hippocampal Growth in Some Subjects Cannot CoRR-OASIS scan–rescan distributions could not be generated for
Entirely be Explained by Scan–Rescan Reliability UCSD and SNT methods; see text for details.

We used literature-reported mean scan–rescan variabili-

ty values for each segmentation algorithm as a threshold
to identify individuals who are plausible candidates for
hippocampal growth, that is, only subjects showing
increases in hippocampal volume above these thresholds
were considered candidate “growers.” Results of this anal-
ysis at 1.5 T are summarized in Table III, divided accord-
ing to QC level. Notably, we found that adjusting for
scan–rescan variability still did not completely exclude
“candidate” hippocampal growers by each method, even
among those subjects who passed stringent across-method
QC. As an example, by the SNT algorithm at 1.5 T, fully
13% of MCI patients and 22% of NC subjects demonstrat-
ed average hippocampal growth greater than scan–rescan
variability threshold at the highest QC level.
Comparisons between the scan–rescan distributions
obtained from FSL, Freesurfer, and MAGeT Brain (using
the CoRR-OASIS dataset), and the 1-year ADNI atrophy
rate distributions are shown in Figures 4 and 5. By all
three algorithms, there were still subjects who exhibited
hippocampal growth at 1 year that exceeded the 90th per-
centile of the right tail of the scan–rescan distribution. This
observation held true even at the most stringent QC level.
Subjects Showing Hippocampal Growth are not
Concordant Across Segmentation Algorithms
Despite unexpectedly large numbers of hippocampal
growers across diagnostic groups and segmentation algo-
rithms, very few individual candidate “growers” were iden-
tified consistently across multiple algorithms (Fig. 3 and
Supporting Information, Fig. 3). At 1.5 T, among subjects
passing first-pass QC, a total of only 7 subjects (0 AD, 3 MCI,
and 4 NC) demonstrated mean hippocampal growth across
all methods. By contrast, on its own, each individual algo-
rithm in isolation found from as low as 76 (UCSD) to as high
as 180 (MAGeT) candidate hippocampal “growers.” Similar-
ly, considering only those subjects who passed stringent
across-method QC, a total of only 3 (0 AD, 1 MCI, and 2 NC)
subjects demonstrated mean hippocampal growth by all
algorithms. Yet, between 22 and 63 subjects demonstrated
hippocampal growth when any given algorithm was consid-
ered on its own at the across-method QC level.
Segmentation Algorithms are Poorly Concordant
in Their Estimates of Hippocampal Volume
Change Over Time
To further characterize the degree of concordance—or
lack thereof—in hippocampal atrophy rates across algo-
rithms, we used the intraclass correlation coefficient (ICC) as
a summary statistic. We computed ICC with a two-way,
mixed-model for a single measure using a consistency agree-
ment definition. As shown in Table (I and IV), ICC values
suggested good inter-algorithm reliability (i.e., ICC > 0.7) for
baseline hippocampal volume measurements, irrespective
of field strength and QC level. However, ICC values for %
hippocampal volume change over 1 year were much lower
(ICC range 0.05–0.319), and well below the threshold for
acceptable inter-algorithm reliability. Neither increased field
strength nor more stringent QC appreciably improved ICC
for % hippocampal volume change. In summary, the five
segmentation algorithms we assessed were very poorly

r 2885 r
 r Sankar et al. r

Figure 4.
Comparison of CoRR-OASIS test–retest distributions to ADNI 1-year hippocampal atrophy
rates, at (a) first pass and (b) across-method QC. Accompanying right-sided insets show mean
absolute scan–rescan values as determined from the CoRR-OASIS dataset.

concordant in their estimates of 1-year hippocampal atrophy
rate across all diagnostic categories, all field strengths, and
all levels of scan QC.
Diagnostic Classification Varies Based on
Atrophy Rates and Method Chosen
Subjectwise classification across algorithms and classifica-
tion methods is not concordant (Fig 6). All methods perform
well for the NC versus AD case (mean ROC AUC range:
0.61–0.91). UCSD performs the best for this classification
(mean ROC AUC range: 0.84–0.91). FSL, FS, and SNT show
similar performance across classifiers (mean ROC AUC
range > 0.7) except for random forests with FSL and SNT
(mean ROC AUC 5 0.58 and 0.67, respectively). MAGeT per-
forms the worst in this comparison with (mean ROC
AUC < 0.66 across classifiers). All segmentation methods
drop in accuracy substantially when classifying NC versus
MCI (mean ROC AUC range: 0.47–0.73) with the majority of
ROC values being <0.70. There is no clearly superior algo-
rithm in this comparison; however, SNT performs least
accurately (mean ROC AUC range: 0.47–54). When classify-
ing NC versus AD, a similar pattern is observed. Classifica-
tion accuracies range from 0.36–0.77 with the majority of
values <0.70. MAGeT achieves the lowest accuracy (mean
ROC AUC 5 0.36) with logistic regression, but performs

r 2886 r
 Figure 5.
ADNI 1-year hippocampal atrophy distributions thresholded at the 90th percentile of the CoRR-
OASIS absolute test–retest distributions, for both positive and negative atrophy rates. Within
each graph, top right percentage indicates the 90th percentile threshold used, while percentages
on either side of the distributions indicate the percentage of subjects exhibiting potential growth
atrophy after thresholding. (a) First pass and (b) across-method QC.
 r Sankar et al.
TABLE IV. Inter-rater reliability between hippocampal segmentation algorithms for hippocampal volume and
atrophy rate as measured using the intraclass correlation coefficient (ICC)
Measure QC
1.5 T
Baseline hippocampal volume First pass
Across-method
% change in hippocampal First pass
volume 1 year
Across-method
3T
Baseline hippocampal volume First pass
Across-method
% change in hippocampal First pass
volume 1 year
Across-method
QC 5 quality control level; ICC 5 two-way mixed model, single measures, intraclass correlation coefficient using a consistency agreement
definition.
within the bounds of the other classification techniques
when used with support vector machines or random forests.
FreeSurfer achieves the best accuracy with random forest
(mean ROC 50.77).
DISCUSSION
Motivated by our preliminary observation that within-
subject hippocampal volume increased over 1 year in
some patients—including those diagnosed with AD—in
the ADNI database, we aimed to determine the full extent
of this phenomenon, and the degree to which it could be
explained by sources of variability in automated hippo-
campal segmentation. Our assessment of five different
semi-automated or automated hippocampal segmentation
algorithms—including three (SNT, Freesurfer, and UCSD)
which were used to generate publicly reported hippocam-
pal volume data in ADNI—showed that a considerable
proportion of ADNI patients demonstrated volumetric hip-
pocampal growth over 1 year by all algorithms spanning
AD, MCI, and NC groups. We also closely analyzed hip-
pocampal segmentation quality for each algorithm and
examined concordance between algorithms. We found a
considerable incidence of hippocampal segmentation fail-
ures for each method across all diagnostic groups. Interest-
ingly, even when stringent QC measures were used to
exclude erroneous segmentations, each algorithm still
identified a number of candidate hippocampal “growers.”
Adjustment for scan–rescan reliability error further
reduced the number of candidate hippocampal growers,
but did not eliminate them entirely. Ultimately, we found
that hippocampal growth was inconsistently identified in
the same subjects by all algorithms, and that 1-year hippo-
campal atrophy rate estimates—as opposed to cross-
sectional, single timepoint estimates of baseline hippocam-
pal volume—were poorly concordant across algorithms. In
r 2888
r
Number of
measurements ICC P value
398 0.764 <0.0001
202 0.842 <0.0001
398 0.050 0.373
202 0.319 <0.0001
62 0.800 <0.0001
36 0.828 <0.0001
62 0.281 <0.0001
36 0.149 <0.005
addition, we demonstrate that there is also no concordance
between algorithms at the level of the single subject.
While there have been several previous studies in the
literature addressing issues of error, variability, and reli-
ability in automated hippocampal segmentation [Morey
et al., 2010; Mulder et al., 2014; Mouiha & Duchesne, 2011;
Pipitone et al., 2014], our study is unique for three reasons:
(1) we frame the issue in the context of hippocampal
“growth” over time; (2) we are primarily concerned with
inter-algorithm concordance in hippocampal atrophy rate,
which is a popular longitudinal biomarker for studies of
putative disease-modifying therapies in AD; and (3) we
assessed five separate segmentation algorithms, including,
but not limited to, those algorithms used to generate
hippocampal volume data in ADNI.
To our knowledge, there have to date been no studies
specifically examining the phenomenon of hippocampal
growth over time, either in the setting of AD or in data
obtained from ADNI. A closer look at the literature, how-
ever, does hint at its occurrence without explicit mention.
In one example, Mouiha and Duchesne [2011] examined
hippocampal atrophy rates in the ADNI dataset using
SNT and Freesurfer, and found that for the SNT algo-
rithm, the mean monthly hippocampal atrophy rate
between 6 and 12 months after baseline was greater than
zero for both the left and right hippocampi in HC sub-
jects. Put differently, the hippocampi of healthy subjects,
segmented by SNT, at a group-wide level, demonstrated
a mean increase in volume over a 6 month interval.
Notably—and in keeping with the pattern of nonconcord-
ance we identified—Freesurfer did not find evidence of
groupwide hippocampal growth when applied to the
same group of HC subjects. However, unlike in our
study, these authors did not report the results of QC on
any segmented images, and only considered scans
acquired at 1.5 T.
r
 r Caveats of Longitudinal Automated Hippocampal Volumetry
Figure 6.
Receiver operating curves and area under the curve measures
for ADNI-1 data passing stringent, across method QC for classi-
fication of (A) AD vs NC; (B) MCI vs NC; and (C) MCI vs. AD.
LR_L1 5 Logistic regression with lasso; SVC5 support vector
machine classification; RFC 5 random forest classification.
Scan–rescan reliability error is perhaps the most
straightforward explanation to account for hippocampal
growth we encountered in this study. Repeated MRI
r 2889
r
scanning of the same subject, even under identical scanner
and acquisition parameters (as in ADNI), does not neces-
sarily generate identical images, because of small changes
in the position of the subject’s head within the MRI coil,
as well as instabilities in the magnetic field even on strin-
gently monitored scanners [Morey et al., 2010]. As a result,
hippocampal segmentations may be different between suc-
cessive scans of the same patient even when no actual vol-
ume changes have occurred. Note that, by contrast, the
output of automated hippocampal segmentation algorithms
is thought to be almost perfectly reproducible when applied
to the same scan at different times, whether one uses different
computing platforms, or runs algorithms in parallel on the
same platform [Gronenschild et al., 2012]. Thresholding by
scan–rescan error, however, still did not completely eliminate
hippocampal growers. There are several potential reasons for
this finding. First, it is possible that we underestimated the
degree of scan–rescan variability. For subjects in ADNI, actual
scan–rescan variability for any given algorithm may well be
greater than literature-reported values or our own internally
generated variability distributions, as these values were
obtained using different scanners than in ADNI, and from
different populations free of older subjects or AD patients
(exception is Hsu et al. [2002]). Second, while the effect of the
duration between repeated scans on scan–rescan variability
in hippocampal volume is poorly understood [Holland et al.,
2009; Holland & Dale, 2011], it would seem intuitive that a
longer interval between scans—such as the 12-month interval
in this study—might increase the likelihood of larger inter-
scan variability, thus, the thresholds we identified from
the literature may be underestimates. Finally, it is possible
that scan–rescan variability is simply larger than average
for those specific individual subjects whom we identified as
“candidate” hippocampal growers.
One intriguing alternative to the scan–rescan reliability
explanation is the possibility that hippocampal growth as
we have observed it represents a true biological phenome-
non. The hippocampus has a notable capacity for plasticity
and regeneration, and several quantitative human MRI
studies have suggested that it may enlarge in response to
physical and mental training paradigms, or during recovery
from neurological disease states (see Fotuhi et al. [2012] for
a review). To our knowledge, there is only one published
instance of therapy-related hippocampal enlargement in
AD, reported in a subset of patients with mild AD treated
using deep brain stimulation of the fornix (Sankar et al.,
[2015]). Though these preliminary findings have yet to be
replicated, it is conceivable that in a certain subset of AD or
MCI subjects that hippocampal volume may increase over
time as a compensatory response to Alzheimer’s pathology
or aging. Also, the accumulation of amyloid pathology
within the hippocampi could theoretically produce a spuri-
ous increase in hippocampal volume in selected patients
(analogous to data in some rodent models of AD [Lau et al.,
2008]). Obviously, these possibilities are purely speculative,
and we certainly do not have any convincing evidence from
r
 r Sankar et al.
our data to support their existence. In a set of post-hoc anal-
yses not shown, we could not identify any characteristic
clinical traits or cerebrospinal fluid (CSF) features unique to
hippocampal growers for any given method, including
those who showed growth over and above scan–rescan vari-
ability thresholds. Perhaps more importantly, in the absence
of any real ground truth in hippocampal volume change for
each patient, and because agreement between segmentation
algorithms in identifying hippocampal growers was so
poor, it is a challenge to know in which patients one ought
to even begin looking for evidence of biological hippocam-
pal growth.
A particularly significant finding of our study is that the
automated hippocampal segmentation algorithms we
assessed are in good agreement regarding baseline hippo-
campal volume, but in rather poor agreement regarding the
magnitude and direction of hippocampal volume changes
over time, for subjects in the ADNI dataset. Interestingly,
poor concordance in hippocampal atrophy rate did not
appear to be remedied by increasing field strength (which
should increase signal-to-noise and tissue contrast in the
hippocampal region) nor by applying more stringent QC
measures. Taken together, these data suggest more generally
that a given automated segmentation algorithm may have a
particular bias toward exaggerating or underestimating vol-
ume change relative to other algorithms for a given subject
with a unique hippocampal morphology. This may most
convincingly be explained by a combination of two key fac-
tors. First, different segmentation algorithms use different a
priori anatomical definitions of the hippocampus; as an
example, some algorithms exclude surrounding white mat-
ter such as the alveus or fimbria (Pipitone et al., 2014], while
others incorporate these into the hippocampal volume [Col-
lins and Pruessner, 2010; Pruessner et al., 2002]. This fact
alone would not necessarily worsen interalgorithm concor-
dance if hippocampal atrophy in AD, MCI, and aging were
evenly distributed across the hippocampus. The second key
factor, however, is that hippocampal atrophy may proceed
at different rates within different regions of the hippocam-
pus, as evidenced by recent work using hippocampal sub-
fields as outcome measures [Apostolova et al., 2012; La Joie
et al., 2010; Mueller et al., 2010]. Accordingly, various algo-
rithms may be differentially sensitive to—and their atrophy
rate estimates differentially influenced by—the subject-
specific spatial distribution of atrophy within the hippocam-
pus, leading to significant discrepancies in the calculated
magnitude of longitudinal hippocampal volume change.
Apart from possibly being driven by differences in neuro-
anatomical definitions, the lack of concordance between
atrophy rates may have significant implications in studies of
aging, pathological aging, and in clinical trials using hippo-
campal volume as a primary outcome measure. Given the
confounds over not only the specificity of hippocampal vol-
umes at the individual subject level in a longitudinal setting
but also the concordance between atrophy rates as well, the
choice of algorithm will have a clear effect on outcome
r 2890
r
measures and the acceptable range of values in research
and clinical trial settings. We previously reported the
effects of proportional biases on hippocampal volume esti-
mation in Pipitone et al. [2014], where we observed that
FSL and FreeSurfer, in particular, tended to overestimate
the size of larger hippocampi while underestimating the
size of smaller hippocampi. We noted the opposite, and
more conservative bias, in MAGeT and SNT algorithms.
Thus, the next step, which is outside the scope of this arti-
cle, would be to elucidate if there is a homologous propor-
tional bias related to atrophy rate. Importantly, researchers
may want to consider the use of different algorithms for
different purposes based on the data we have presented
here and depending on the overall endpoints for their
studies or clinical trials.
Overall, our rigorous and comprehensive approach to QC
resulted in the rejection of a large number of hippocampal
segmentations across algorithms, and merits some comment.
For the Freesurfer and UCSD algorithms, we made the deci-
sion to use QC data reported in ADNI, since these data are
publicly accessible. Since many studies consider SNT seg-
mentations to be a “gold standard”—given that they require
observer input and correction during their computation—we
did not consider QC for SNT [Leung et al., 2010; Wolz et al.,
2010a]. For those algorithms lacking QC information in ADNI
(i.e., FSL and MAGeT), we performed a detailed visual
inspection of segmentation quality in each subject. Given the
absence of a standard consensus method to assess the quality
of hippocampal segmentations, we erred on the side of a strict
approach, limiting as much as possible the potential influence
of spurious segmentations on the measured incidence of hip-
pocampal growers. As an example, for the FreeSurfer algo-
rithm, we classified a particular subject as a hippocampal
segmentation failure if that subject showed evidence of seg-
mentation inaccuracies anywhere in the brain across a multi-
tude of cortical or subcortical regions (i.e., if that subject failed
“overall” QC). Our rationale was that evidence of poor tissue
classification or poor delineation of anatomy anywhere in the
brain might introduce doubt about the accuracy of hippocam-
pal segmentations as well. No doubt this contributed to the
high failure rate we found for the FreeSurfer algorithm. Note
that we are neither advocating any particular approach to
assessing hippocampal segmentation quality, nor are we
suggesting that the ideal approach needs to be as stringent as
the one we employed. That being said, our QC data are con-
sistent with recent reports in the literature suggesting that the
influence of failed segmentations may have the effect of
significantly reducing the reproducibility of hippocampal
volume change estimates in longitudinal studies [Mulder
et al., 2014]. Indeed, our findings underscore the need to re-
evaluate the validity of the oft-used argument that it is too
costly to perform rigorous QC of hippocampal segmentations
in large datasets. At the very least, our results would argue
that in studies using ADNI-reported hippocampal values, the
(easily accessed) accompanying QC data ought to be consid-
ered as well.
r
 r Caveats of Longitudinal Automated Hippocampal Volumetry
One open question based on the findings presented in this
article is the ability of these algorithms to classify individuals
by their diagnosis. While this could be an important under-
taking in the context of the hippocampal volume segmenta-
tion, such analyses are a topic of hot debate in the literature
[Coupe et al., 2011a,; Davatzikos et al., 2011; Eskildsen et al.,
2013; Wang et al., 2010]. Such a problem would require sever-
al design choices, which themselves will be controversial,
each of which could easily account for a source of detailed
investigation. For example, should the input data that is used
consist of baseline volumes only, atrophy rates, or volumes
from both time points? Should data that passes QC on a per-
method basis be used or should the dataset consist of only
those subjects that have passed QC across several methods?
The impact of image processing techniques on multi-variate/
machine learning applications is an open question in the liter-
ature that requires further investigation, but is, ultimately,
outside the scope of this article.
Nevertheless, based on our segmentation results, we also
performed classification of the different diagnostic catego-
ries using three commonly used classifiers (logistic regres-
sion with lasso, random forests, and support vector
machines). Other groups have used classification techniques
to better identify diagnostic groups. For example, Wolz
et al [2010b] achieve a correct classification rate of 82%
between AD and NC and they also demonstrate they can
identify MCI patients who progress to AD (with an accura-
cy of 64%) based only on 1-year atrophy rates. Although
we did not perform this latter comparison, it could be fur-
ther explored using the framework in this manuscript.
However, given the large number of comparisons, segmen-
tation algorithms, and classifiers already explored in this
manuscript, it would have been difficult to treat the latter
question (MCI-to-AD conversion) appropriately while main-
taining equivalent datasets across methods (given the sig-
nificant issues with quality control that we observed). It is
important to note that there are other studies that examine
the utility of hippocampal volume as a biomarker based on
the relationship between hippocampal atrophy and estab-
lished biomarkers related to AD pathophysiology. These
include the observations that longitudinal atrophy rates
measured using surface-based [Morra et al., 2008] and volu-
metric techniques [Schuff et al., 2009] have a strong rela-
tionship with known AD-specific risk factors (such as being
positive for apolipoprotein e4 isoform and low education
status) and CSF biomarkers.
Another open question that remains is how to deal with
the lack of concordance between methodologies. While
many groups have started to move to standardized datasets
and practices in evaluating hippocampal and hippocampal
subfield volumes, further investigation may be necessary
[Boccardi et al., 2013a; Schoemaker et al., 2016; Yushkevich
et al., 2015]. Heterogeneity in scientific methodology is
acceptable, so long as methodologies have been stringently
validated (as is the case with algorithms that have been eval-
uated here). Furthermore, our understanding of group
r 2891
r
effects certainly benefits from many groups asking similar
questions but answering them with different tools. While
these algorithms show accurate results at the group level
(significant cross-sectional and longitudinal differences), it
is clear that their precision (i.e., treatment of single subjects,
atrophy rates) varies substantially. This is important and
requires further replication and research with the use of
open-source software and shared datasets.
It may also be worth considering the differences between
the design of the different methods. In this study, two of the
methods that were used were explicitly designed for the
analyses of longitudinal datasets. Previous work by Jovovich
et al. [2014] examined the reliability of the FreeSurfer cross-
sectional and longitudinal processing streams in a multisite
setting and demonstrated far more variability in the cross-
sectional scheme across sites for hippocampal volumes.
These results are reflected in our results as well, after strin-
gent QC of all methods is performed, we observe a slightly
larger number of candidate “growers” in methods not opti-
mized for longitudinal data. The UCSD method appears to
perform better in this regard and shows a small number of
growers across QC methods. Nontheless, it is somewhat
puzzling why in some instances FreeSurfer demonstrates a
substantial number of candidate growers (>7% in some
cases; see Table III). It is possible that this can be attributed
to older imaging (1.5 T) and image processing pipelines
(older FreeSurfer versions) that were used in the processing
of the data. It remains to be seen if newer versions of the
FreeSurfer pipeline may behave differently with these data.
One limitation of our study is that we did not compare the
incidence of hippocampal growth in ADNI by manual trac-
ing to that observed with automated segmentation algo-
rithms. Manual tracing by trained observers is considered
by some to be more accurate than automated methods in
determining either hippocampal volume or atrophy rate
[Barnes et al., 2008; Boccardi et al., 2011a, 2013a; Nestor
et al., 2012; Pipitone et al., 2014; Pruessner et al., 2000], and
segmentations generated manually are usually considered
the gold standard against which automated methods are
validated [Mulder et al., 2014]. In addition, manual tracing
is less susceptible to gross segmentation errors [Morey et al.,
2010]. At first glance, therefore, manual tracing would
appear better suited to predicting the true incidence of hip-
pocampal growers—if they actually exist—provided manual
segmentations could actually be completed on all subjects in
ADNI. Unfortunately, doing so would most likely be prohib-
itively time-consuming. Even if it could be done, it is not
necessarily true that the results obtained would be more val-
id than for automated segmentation. First, unlike automated
segmentation algorithms, the reproducibility of any hippo-
campal manual tracing protocol is influenced by intra-rater
reliability, as observer interpretation of identical imaging
data may drift over time and due to human factors such as
fatigue [Morey et al., 2010]. Second, inter-rater reliability for
the same protocol may vary even between expert observers
[Chupin et al., 2009; Pipitone et al., 2014]. Third, in the
r
 r Sankar et al.
absence of widespread adoption of a harmonized protocol,
there are at least as many manual protocols as there are
automated segmentation algorithms [Boccardi et al., 2013a,
2013b]. Accordingly, depending on the specific hippocampal
boundaries used in each protocol, the subject-specific spatial
distribution of atrophy within the hippocampus may influ-
ence inter-protocol concordance in hippocampal atrophy
rates in a manner similar to that observed with automated
algorithms.
A second limitation concerns the results reported by thresh-
olding for true growth using literature-based values. These
values represent a mean error on the reliability of the mea-
surement of hippocampal volume; consequently, any thresh-
old defined by this method is liberal as it is possible that
scan–rescan reliability for any given subject may exceed the
group mean. Indeed, we would caution against overinterpret-
ing results that exceed mean scan–rescan reliability thresholds
in cases where information about a single subject is desired.
Nonetheless, our results based on this type of thresholding
still demonstrate a lack of concordance between many com-
monly used algorithms in the literature. A final limitation per-
tains to our analysis of atrophy rates. While it is possible that
atrophy rates may not theoretically conform to a normal dis-
tribution (as atrophy rate is the ratio of normally distributed
variables), this is not what we observe in our data, (except for
UCSD data at 1.5 T), we chose to use Gaussian statistics for
analysis of atrophy rates.
Taken together, our findings raise some concerns in the use
of hippocampal volume as a biomarker in studies of AD and
other neurodegenerative disorders. First, QC may have a
major impact regardless of segmentation algorithm, field
strength, diagnosis, and hippocampal boundary definitions.
The high rate of QC failures in this study brings into question
whether current and prior “black-box” approaches to hippo-
campal volumetry applied in certain studies and clinical tri-
als, where patient MRIs are analyzed using either commercial
or publically available software without any standardized
QC measures reported, can be entirely trusted to yield mean-
ingful results. Second, our data indicate that relative longitu-
dinal hippocampal volume change, calculated in an
automated manner, may be a suboptimal biomarker to track
disease progression or response to therapy. This appears to
hold true even for algorithms that have been optimized for
longitudinal volumetric processing, such as FreeSurfer and
the UCSD method; we found these algorithms to be poorly
concordant with one another and with alternate algorithms.
Our data further suggest that it may be ill-advised to trust
normative or pathological rates of hippocampal volume
change derived only a single algorithm; conversely, it is pos-
sible that the use of several different algorithms could, at
worst, produce altogether divergent results.
CONCLUSION
In an assessment of five different automated or semi-
automated segmentation algorithms designed to measure
r 2892
r
hippocampal volume from structural MRI data, we found
an unexpectedly high incidence of subjects in the ADNI
database who demonstrated hippocampal growth over
time. For no individual algorithm could this counterintui-
tive finding be entirely explained by gross segmentation
errors, scan–rescan variability, or field strength of MRI
acquisition. Furthermore, algorithms did not consistently
identify the same subjects as hippocampal growers, and
more generally our analysis revealed poor concordance
between algorithms in their estimates of the magnitude
and direction of hippocampal volume change over time,
which precluded a meaningful analysis of whether hippo-
campal growth could be a true biological phenomenon.
These findings suggest that, in patients with either AD or
MCI, or age-matched elderly controls, longitudinal hippo-
campal volume change should be interpreted with consid-
erable caution as a biomarker at the individual subject
level, and may have important limitations at a group-wide
level as well.
ACKNOWLEDGMENTS
T.S. is supported by a Canadian Institutes of Health
Research (CIHR) fellowship award. A.N.V. is funded by the
Canadian Institutes of Health Research (CIHR), National
Alliance for Research on Schizophrenia and Depression
(NARSAD), Ontario Mental Health Foundation (OMHF),
and CAMH Foundation (Koerner New Scientist Program
and Paul Garfinkel New Investigator Catalyst Fund). A.M.L.
is a Canada Research Chair in Neuroscience and is sup-
ported by the R.R. Tasker Chair in Functional Neurosurgery.
A.M.L. is a consultant to Medtronic, St Jude, and Boston Sci-
entific. A.M.L. serves on the scientific advisory board of Cer-
egene, Codman, Neurophage, Aleva, and Alcyone Life
Sciences. A.M.L. is a co-founder of Functional Neuromodu-
lation Inc. and holds intellectual property in the field of
Deep Brain Stimulation. M.M.C. is supported by funding
from the W. Garfield Weston Foundation, Michael J. Fox
Foundation, Alzheimer’s Society, National Sciences and
Engineering Research Council of Canada, and Canadian
Institutes of Health Research. MR image processing compu-
tations were performed on the GPC supercomputer at the
SciNet HPC Consortium. SciNet is funded by the Canada
Foundation for Innovation under the auspices of Compute
Canada; the Government of Ontario; Ontario Research Fund
- Research Excellence; and the University of Toronto.
Data collection and sharing for this project was funded by
the Alzheimer’s Disease Neuroimaging Initiative (ADNI)
and DOD ADNI. ADNI is funded by the National Institute
on Aging, the National Institute of Biomedical Imaging
and Bioengineering, and through generous contributions
from the following: AbbVie, Alzheimer’s Association; Alz-
heimer’s Drug Discovery Foundation; Araclon Biotech;
BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company;
CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals,
Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La
Roche Ltd and its affiliated company Genentech, Inc.;
r
 r Caveats of Longitudinal Automated Hippocampal Volumetry
Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer
Immunotherapy Research & Development, LLC.; Johnson
& Johnson Pharmaceutical Research & Development LLC.;
Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diag-
nostics, LLC.; NeuroRx Research; Neurotrack Technolo-
gies; Novartis Pharmaceuticals Corporation; Pfizer Inc.;
Piramal Imaging; Servier; Takeda Pharmaceutical Compa-
ny; and Transition Therapeutics. The Canadian Institutes
of Health Research is providing funds to support ADNI
clinical sites in Canada. Private sector contributions are
facilitated by the Foundation for the National Institutes of
Health (www.fnih.org). The grantee organization is the
Northern California Institute for Research and Education,
and the study is coordinated by the Alzheimer’s Thera-
peutic Research Institute at the University of Southern Cal-
ifornia. ADNI data are disseminated by the Laboratory for
Neuro Imaging at the University of Southern California.
Data collected for the OASIS dataset were performed
using support obtained from the NIH grants.
REFERENCES
Amaral RS, Park MT, Devenyi GA, Lynn V, Pipitone J,
Winterburn J, Chavez S, Schira M, Lobaugh NJ, Voineskos
AN, Pruessner JC, Chakravarty MM Alzheimer’s Disease Neu-
roImaging Initiative (2016): Manual segmentation of the fornix,
fimbria, and alveus on high-resolution 3T MRI: Application via
fully-automated mapping of the human memory circuit white
and grey matter in healthy and pathological aging. Neuro-
image epub. doi: 10.1016/j.neuroimage.2016.10.027.
Apostolova LG, Morra JH, Green AE, Hwang KS, Avedissian C,
Woo E, Cummings JL, Toga AW, Jack CR Jr, Weiner MW,
Thompson PM (2010): Automated 3D mapping of baseline and
12-month associations between three verbal memory measures
and hippocampal atrophy in 490 ADNI subjects. Neuroimage
51:488–499.
Apostolova LG, Green AE, Babakchanian S, Hwang KS, Chou YY,
Toga AW, Thompson PM (2012): Hippocampal atrophy and
ventricular enlargement in normal aging, mild cognitive
impairment (MCI), and Alzheimer Disease. Alzheimer Dis
Assoc Disord 26:17–27.
Avants BB, Epstein CL, Grossman M, Gee JC (2008): Symmetric dif-
feomorphic image registration with cross-correlation: Evaluating
automated labeling of elderly and neurodegenerative brain. Med
Image Anal 12:26–41.
Barnes J, Foster J, Boyes RG, Pepple T, Moore EK, Schott JM, Scahill RI,
Fox NC (2008): A comparison of methods for the automated calcu-
lation of volumes and atrophy rates in the hippocampus. Neuro-
image 40:1655–1671.
Barnes J, Ourselin S, Fox NC (2009): Clinical application of mea-
surement of hippocampal atrophy in degenerative dementias.
Hippocampus 19:510–516.
Boccardi M, Bocchetta M, Apostolova LG, Preboske G, Robitaille N,
Pasqualetti P, Collins LD, Duchesne S, Jack CR Jr, Frisoni GB
(2013a): Establishing magnetic resonance images orientation for the
EADC-ADNI manual hippocampal segmentation protocol.
J Neuroimaging. doi:10.1111/jon.12065
Boccardi M, Bocchetta M, Ganzola R, Robitaille N, Redolfi A,
Duchesne S, Jack CR Jr, Frisoni GB (2013b): Operationalizing
r 2893
r
protocol differences for EADC-ADNI manual hippocampal seg-
mentation. Alzheimers Dement doi:10.1016/j.jalz.2013.03.001
Boccardi M, Bocchetta M, Ganzola R, Robitaille N, Redolfi A,
Bartzokis G, Camicioli R, Csernansky J , De Leon M, DeToledo-
Morrell M, Killiany M, Lehericy S, Pantel J, Pruessner J,
Soininen H, Watson C, Duchesne S, Jack C, Frisoni G (2011a):
Estimating the impact of differences among protocols for manu-
al hippocampal segmentation on Alzheimer’s disease-related
atrophy: Preparatory phase for a harmonized protocol.
Alzheimers Dement 7:S205–S206.
Boccardi M, Ganzola R, Bocchetta M, Pievani M, Redolfi A,
Bartzokis G, Camicioli R, Csernansky JG, de Leon MJ,
deToledo-Morrell L, Killiany RJ, Lehericy S, Pantel J, Pruessner
JC, Soininen H, Watson C, Duchesne S, Jack CR Jr, deToledo-
Morrell L (2011b): Survey of protocols for the manual
segmentation of the hippocampus: Preparatory steps towards
a joint EADC-ADNI harmonized protocol. J Alzheimers Dis 26:
61–75.
Braak H, Braak E (1991): Neuropathological stageing of
Alzheimer-related changes. Acta Neuropathol 82:239–259.
Braak H, de Vos RA, Jansen EN, Bratzke H, Braak E (1998): Neu-
ropathological hallmarks of Alzheimer’s and Parkinson’s dis-
eases. Prog Brain Res 117:267–285.
Cavedo E, Pievani M, Boccardi M, Galluzzi S, Bocchetta M,
Bonetti M, Thompson PM, Frisoni GB (2014): Medial temporal
atrophy in early and late-onset Alzheimer’s disease. Neurobiol
Aging doi:10.1016/j.neurobiolaging.2014.03.009
Chakravarty MM, Steadman P, van Eede MC, Calcott RD, Gu V,
Shaw P, Raznahan A, Collins DL, Lerch JP (2013): Performing
label-fusion-based segmentation using multiple automatically
generated templates. Hum Brain Mapp 34:2635–2654.
Chupin M, Gerardin E, Cuingnet R, Boutet C, Lemieux L,
Lehericy S, Benali H, Garnero L, Colliot O (2009): Fully auto-
matic hippocampus segmentation and classification in Alz-
heimer’s disease and mild cognitive impairment applied on
data from ADNI. Hippocampus 19:579–587.
Collins DL, Pruessner JC (2010): Towards accurate, automatic seg-
mentation of the hippocampus and amygdala from MRI by aug-
menting ANIMAL with a template library and label fusion.
Neuroimage 52:1355–1366.
Coupe P, Eskildsen SF, Manjon JV, Fonov V, Collins DL (2011a):
Simultaneous segmentation and grading of hippocampus for
patient classification with Alzheimer’s disease. Med Image
Comput Comput Assist Interv 14:149–157.
Coupe P, Eskildsen SF, Manjon JV, Fonov V, Collins DL (2011b):
Simultaneous segmentation and grading of anatomical struc-
tures for patient’s classification: Application to Alzheimer’s
disease. Neuroimage doi:10.1016/j.neuroimage.2011.10.080
Coupe P, Fonov VS, Bernard C, Zandifar A, Eskildsen SF, Helmer
C, Manj on JV, Amieva H, Dartigues JF, Allard M, Catheline G,
Collins DL (2015): Detection of Alzheimer’s disease signature
in MR images seven years before conversion to dementia:
Toward an early individual prognosis. Hum Brain Mapp doi:
10.1002/hbm.22926
Davatzikos C, Bhatt P, Shaw LM, Batmanghelich KN, Trojanowski
JQ (2011): Prediction of MCI to AD conversion, via MRI, CSF
biomarkers, and pattern classification. Neurobiol Aging 32:
2322.e19–2327.
Eskildsen SF, Coupe P, Garcia-Lorenzo D, Fonov V, Pruessner JC,
Collins DL (2013): Prediction of Alzheimer’s disease in subjects
with mild cognitive impairment from the ADNI cohort using
patterns of cortical thinning. Neuroimage 65:511–521.
r
 r Sankar et al.
Fischl B, van der Kouwe A, Destrieux C, Halgren E, Segonne F,
Salat DH, Busa E, Seidman LJ, Goldstein J, Kennedy D,
Caviness V, Makris N, Rosen B, Dale AM (2004): Automatical-
ly parcellating the human cerebral cortex. Cereb Cortex 14:
11–22.
Fischl B, Salat DH, Busa E, Albert M, Dieterich M, Haselgrove C,
van der Kouwe A, Killiany R, Kennedy D, Klaveness S,
Montillo A, Makris N, Rosen B, Dale AM (2002): Whole brain
segmentation: Automated labeling of neuroanatomical struc-
tures in the human brain. Neuron 33:341–355.
Fotuhi M, Do D, Jack C (2012): Modifiable factors that alter the
size of the hippocampus with ageing. Nat Rev Neurol 8:
189–202.
Frisoni GB, Fox NC, Jack CRJ, Scheltens P, Thompson PM (2010):
The clinical use of structural MRI in Alzheimer disease. Nat
Rev Neurol 6:67–77.
Gronenschild EH, Habets P, Jacobs HI, Mengelers R, Rozendaal N,
van Os J, Marcelis M (2012): The effects of FreeSurfer version,
workstation type, and Macintosh operating system version on
anatomical volume and cortical thickness measurements. PLoS
One 7:e38234.
Holland D, Brewer JB, Hagler DJ, Fennema-Notestine C, Dale AM
(2009): Subregional neuroanatomical change as a biomarker for
Alzheimer’s disease. Proc Natl Acad Sci USA 106:20954–20959.
Holland D, Dale AM (2011): Nonlinear registration of longitudinal
images and measurement of change in regions of interest. Med
Image Anal 15:489–497.
Holland D, McEvoy LK, Dale AM (2012): Unbiased comparison
of sample size estimates from longitudinal structural mea-
sures in ADNI. Hum Brain Mapp 33:2586–2602.
Hsu YY, Schuff N, Du AT, Mark K, Zhu X, Hardin D, Weiner
MW (2002): Comparison of automated and manual MRI volu-
metry of hippocampus in normal aging and dementia. J Magn
Reson Imaging 16:305–310.
Jack CR Jr, Bernstein MA, Fox NC, Thompson P, Alexander G,
Harvey D, Borowski B, Britson PJ, L Whitwell J, Ward C, Dale
AM, Felmlee JP, Gunter JL, Hill DL, Killiany R, Schuff N, Fox-
Bosetti S, Lin C, Studholme C, DeCarli CS, Krueger G, Ward
HA, Metzger GJ, Scott KT, Mallozzi R, Blezek D, Levy J,
Debbins JP, Fleisher AS, Albert M, Green R, Bartzokis G,
Glover G, Mugler J, Weiner MW (2008): The Alzheimer’s Dis-
ease Neuroimaging Initiative (ADNI): MRI methods. J Magn
Reson Imaging 27:685–691.
Jenkinson M, Beckmann CF, Behrens TE, Woolrich MW, Smith
SM (2012): FSL. Neuroimage 62:782–790.
Jovicich J, Marizzoni M, Bosch B, Bartres-Faz D, Arnold J,
Benninghoff J, Wiltfang J, Roccatagliata L, Picco A, Nobili F,
Blin O, Bombois S, Lopes R, Bordet R, Chanoine V, Ranjeva JP,
Didic M, Gros-Dagnac H, Payoux P, Zoccatelli G, Alessandrini
F, Beltramello A, Bargallû˚ N, Ferretti A, Caulo M, Aiello M,
Ragucci M, Soricelli A, Salvadori N, Tarducci R, Floridi P,
Tsolaki M, Constantinidis M, Drevelegas A, Rossini PM, Marra
C, Otto J, Reiss-Zimmermann M, Hoffmann KT, Galluzzi S,
Frisoni GB, PharmaCog C (2014): Multisite longitudinal reli-
ability of tract-based spatial statistics in diffusion tensor imag-
ing of healthy elderly subjects. Neuroimage 101:390–403.
La Joie R, Fouquet M, Mezenge F, Landeau B, Villain N, Mevel K,
Pelerin A, Eustache F, Desgranges B, Chetelat G (2010): Differ-
ential effect of age on hippocampal subfields assessed using a
new high-resolution 3T MR sequence. Neuroimage 53:506–514.
Lau JC, Lerch JP, Sled JG, Henkelman RM, Evans AC, Bedell BJ
(2008): Longitudinal neuroanatomical changes determined by
r 2894
r
deformation-based morphometry in a mouse model of Alz-
heimer’s disease. Neuroimage 42:19–27.
Lerch JP, Pruessner J, Zijdenbos AP, Collins DL, Teipel SJ, Hampel
H, Evans AC (2008): Automated cortical thickness measure-
ments from MRI can accurately separate Alzheimer’s patients
from normal elderly controls. Neurobiol Aging 29:23–30.
Leung KK, Barnes J, Ridgway GR, Bartlett JW, Clarkson MJ,
Macdonald K, Schuff N, Fox NC, Ourselin S (2010): Automat-
ed cross-sectional and longitudinal hippocampal volume mea-
surement in mild cognitive impairment and Alzheimer’s
disease. Neuroimage 51:1345–1359.
McLaren DG, Sreenivasan A, Diamond EL, Mitchell MB, Van Dijk
KR, Deluca AN, O’Brien JL, Rentz DM, Sperling RA, Atri A
(2012): Tracking cognitive change over 24 weeks with longitudi-
nal functional magnetic resonance imaging in Alzheimer’s dis-
ease. Neurodegener Dis 9:176–186.
Mielke MM, Okonkwo OC, Oishi K, Mori S, Tighe S, Miller MI,
Ceritoglu C, Brown T, Albert M, Lyketsos CG (2012): Fornix
integrity and hippocampal volume predict memory decline
and progression to Alzheimer’s disease. Alzheimers Dement 8:
105–113.
Miller M, Banerjee A, Christensen G, Joshi S, Khaneja N,
Grenander U, Matejic L (1997): Statistical methods in computa-
tional anatomy. Stat Methods Med Res 6:267–299.
Morey RA, Selgrade ES, Wagner HR, Huettel SA, Wang L,
McCarthy G (2010): Scan-rescan reliability of subcortical brain
volumes derived from automated segmentation. Hum Brain
Mapp 31:1751–1762.
Morra JH, Tu Z, Apostolova LG, Avedissian C, Madsen SK,
Parikshak N, Hua X, Toga AW, Jack CR, Jr, Weiner MW,
Thompson PM (2008): Alzheimer’s disease neuroimaging initia-
tive. NeuroImage 43:59–68.
Mouiha A, Duchesne S (2011): Hippocampal atrophy rates in Alz-
heimer’s disease: Automated segmentation variability analysis.
Neurosci Lett Retrieved from http://www.sciencedirect.com/
science/article/pii/S0304394011002564
Mueller SG, Weiner MW, Thal LJ, Petersen RC, Jack CR, Jagust
W, Trojanowski JQ, Toga AW, Beckett L (2005): Ways toward
an early diagnosis in Alzheimer’s disease: The Alzheimer’s
Disease Neuroimaging Initiative (ADNI). Alzheimers Dement
1:55–66.
Mueller SG, Schuff N, Yaffe K, Madison C, Miller B, Weiner MW
(2010): Hippocampal atrophy patterns in mild cognitive
impairment and Alzheimer’s disease. Hum Brain Mapp 31:
1339–1347.
Mulder ER, de Jong RA, Knol DL, van Schijndel RA, Cover KS,
Visser PJ, Barkhof F, Vrenken H (2014): Hippocampal volume
change measurement: Quantitative assessment of the reproduc-
ibility of expert manual outlining and the automated methods
FreeSurfer and FIRST. Neuroimage 92:169–181.
Nestor SM, Gibson E, Gao FQ, Kiss A, Black SE (2012): A direct
morphometric comparison of five labeling protocols for multi-
atlas driven automatic segmentation of the hippocampus in
Alzheimer’s disease. Neuroimage 66C:50–70.
Patenaude B, Smith SM, Kennedy DN, Jenkinson M (2011): A
Bayesian model of shape and appearance for subcortical brain
segmentation. Neuroimage 56:907–922.
Pipitone J, Park MT, Winterburn J, Lett TA, Lerch JP, Pruessner
JC, Lepage M, Voineskos AN, Chakravarty MM (2014): Multi-
atlas segmentation of the whole hippocampus and subfields
using multiple automatically generated templates. Neuroimage
101:494–512.
r
 r Caveats of Longitudinal Automated Hippocampal Volumetry
Pruessner JC, Kohler S, Crane J, Pruessner M, Lord C, Byrne A,
Kabani N, Collins DL, Evans AC (2002): Volumetry of temporopo-
lar, perirhinal, entorhinal and parahippocampal cortex from
high-resolution MR images: Considering the variability of the col-
lateral sulcus. Cereb Cortex 12:1342–1353.
Pruessner JC, Li LM, Serles W, Pruessner M, Collins DL, Kabani N,
Lupien S, Evans AC (2000): Volumetry of hippocampus and
amygdala with high-resolution MRI and three-dimensional anal-
ysis software: Minimizing the discrepancies between laborato-
ries. Cereb Cortex 10:433–442.
Reuter M, Schmansky NJ, Rosas HD, Fischl B (2012): Within-sub-
ject template estimation for unbiased longitudinal image analy-
sis. Neuroimage 61:1402–1418.
Sabuncu MR, Buckner RL, Smoller JW, Lee PH, Fischl B, Sperling
RA (2012): The association between a polygenic Alzheimer
score and cortical thickness in clinically normal subjects. Cereb
Cortex 22:2653–2661.
Sabuncu MR, Desikan RS, Sepulcre J, Yeo BT, Liu H, Schmansky
NJ, Reuter M, Weiner MW, Buckner RL, Sperling RA, Fischl B
(2011): The dynamics of cortical and hippocampal atrophy in
Alzheimer disease. Arch Neurol 68:1040–1048.
Sanchez-Benavides G, Pena-Casanova J, Casals-Coll M, Gramunt
N, Molinuevo JL, Gomez-Anson B, Aguilar M, Robles A,
Ant unez C, Martınez-Parra C, Frank-Garcıa A, Fern andez-
Martınez M, Blesa R (2014): Cognitive and neuroimaging pro-
files in mild cognitive impairment and Alzheimer’s disease:
Data from the Spanish Multicenter Normative Studies (NEU-
RONORMA project). J Alzheimers Dis doi:10.3233/JAD-132186
Sankar T, Chakravarty MM, Bescos A, Lara M, Obuchi T, Laxton
AW, McAndrews MP, Tang-Wai DF, Workman CI, Smith GS,
Lozano AM (2015): Deep Brain Stimulation Influences Brain
Structure in Alzheimer’s Disease. Brain Stimulation 8:645–654.
Schoemaker D, Buss C, Head K, Sandman CA, Davis EP,
Chakravarty MM, Gauthier S, Pruessner JC (2016): Hippocam-
pus and amygdala volumes from magnetic resonance images
in children: Assessing accuracy of FreeSurfer and FSL against
manual segmentation. Neuroimage 129:1–14.
Schuff N, Woerner N, Boreta L, Kornfield T, Shaw LM,
Trojanowski JQ, Thompson PM, Jack CR Jr, Weiner MW; Alz-
heimer’s Disease NeuroImaging Initiative (2009): MRI of hip-
pocampal volume loss in early Alzheimer’s disease in relation
to ApoE genotype and biomakers. 132(Pt 4):1067–1077.
Small SA, Duff K (2008): Linking Abeta and tau in late-onset Alz-
heimer’s disease: A dual pathway hypothesis. Neuron 60:
534–542.
Stein JL, Medland SE, Vasquez AA, Hibar DP, Senstad RE,
Winkler AM, Toro R, Appel K, Bartecek R, Bergmann Ø,
Bernard M, Brown AA, Cannon DM, Chakravarty MM,
Christoforou A, Domin M, Grimm O, Hollinshead M, Holmes
AJ, Homuth G, Hottenga JJ, Langan C, Lopez LM, Hansell
NK, Hwang KS, Kim S, Laje G, Lee PH, Liu X, Loth E,
Lourdusamy A, Mattingsdal M, Mohnke S, Maniega SM, Nho
K, Nugent AC, O’Brien C, Papmeyer M, P€ utz B, Ramasamy A,
Rasmussen J, Rijpkema M, Risacher SL, Roddey JC, Rose EJ,
Ryten M, Shen L, Sprooten E, Strengman E, Teumer A,
Trabzuni D, Turner J, van Eijk K, van Erp TG, van Tol MJ,
Wittfeld K, Wolf C, Woudstra S, Aleman A, Alhusaini S,
Almasy L, Binder EB, Brohawn DG, Cantor RM, Carless MA,
Corvin A, Czisch M, Curran JE, Davies G, de Almeida MA,
Delanty N, Depondt C, Duggirala R, Dyer TD, Erk S,
Fagerness J, Fox PT, Freimer NB, Gill M, G€ oring HH, Hagler
DJ, Hoehn D, Holsboer F, Hoogman M, Hosten N, Jahanshad
r 2895
r
N, Johnson MP, Kasperaviciute D, Kent JW Jr, Kochunov P,
Lancaster JL, Lawrie SM, Liewald DC, Mandl R, Matarin M,
Mattheisen M, Meisenzahl E, Melle I, Moses EK, M€ uhleisen
TW, Nauck M, N€ othen MM, Olvera RL, Pandolfo M, Pike GB,
Puls R, Reinvang I, Renterıa ME, Rietschel M, Roffman JL,
Royle NA, Rujescu D, Savitz J, Schnack HG, Schnell K, Seiferth
N, Smith C, Steen VM, Valdes Hern andez MC, Van den
Heuvel M, van der Wee NJ, van Haren NE, Veltman JA,
V€olzke H, Walker R, Westlye LT, Whelan CD, Agartz I,
Boomsma DI, Cavalleri GL, Dale AM, Djurovic S, Drevets WC,
Hagoort P, Hall J, Heinz A, Jack CR Jr, Foroud TM, Le Hellard
S, Macciardi F, Montgomery GW, Poline JB, Porteous DJ,
Sisodiya SM, Starr JM, Sussmann J, Toga AW, Veltman DJ,
Walter H, Weiner MW; Alzheimer’s Disease Neuroimaging
Initiative.; EPIGEN Consortium.; IMAGEN Consortium.; Sag-
uenay Youth Study Group., Bis JC, Ikram MA, Smith AV,
Gudnason V, Tzourio C, Vernooij MW, Launer LJ, DeCarli C,
Seshadri S; Cohorts for Heart and Aging Research in Genomic
Epidemiology Consortium., Andreassen OA, Apostolova LG,
Bastin ME, Blangero J, Brunner HG, Buckner RL, Cichon S,
Coppola G, de Zubicaray GI, Deary IJ, Donohoe G, de Geus
EJ, Espeseth T, Fern andez G, Glahn DC, Grabe HJ, Hardy J,
Hulshoff Pol HE, Jenkinson M, Kahn RS, McDonald C,
McIntosh AM, McMahon FJ, McMahon KL, Meyer-Lindenberg
A, Morris DW, M€ uller-Myhsok B, Nichols TE, Ophoff RA,
Paus T, Pausova Z, Penninx BW, Potkin SG, S€ amann PG,
Saykin AJ, Schumann G, Smoller JW, Wardlaw JM, Weale ME,
Martin NG, Franke B, Wright MJ Thompson PM (2012): Identi-
fication of common variants associated with human hippocam-
pal and intracranial volumes. Nat Genet 44:552–561.
Treadway MT, Waskom ML, Dillon DG, Holmes AJ, Park MT,
Chakravarty MM, Dutra SJ, Polli FE, Iosifescu DV, Fava M,
Gabrieli JD, Pizzagali DA (2015): Illness progression, recent
stress, and morphometry of the hippocampal subfields and
medial prefrontal cortex in major depression. Biological Psy-
chiatry 77:265–294.
Wang Y, Fan Y, Bhatt P, Davatzikos C (2010): High-dimensional pat-
tern regression using machine learning: From medical images to
continuous clinical variables. Neuroimage 50:1519–1535.
Watson C, Cendes F, Fuerst D, Dubeau F, Williamson B, Evans A,
Andermann F (1997): Specificity of volumetric magnetic reso-
nance imaging in detecting hippocampal sclerosis. Arch Neurol
54:67–73.
Weiner MW, Veitch DP, Aisen PS, Beckett LA, Cairns NJ, Green
RC, Harvey D, Jack CR, Jagust W, Liu E, Morris JC, Petersen
RC, Saykin AJ, Schmidt ME, Shaw L, Siuciak JA, Soares H,
Toga AW, Trojanowski JQ (2012): The Alzheimer’s Disease
Neuroimaging Initiative: A review of papers published since
its inception. Alzheimers Dement 8:S1–68.
Winterburn JL, Pruessner JC, Chavez S, Schira MM, Lobaugh NJ,
Voineskos AN, Chakravarty MM (2013): A novel in vivo atlas
of human hippocampal subfields using high-resolution 3 T
magnetic resonance imaging. Neuroimage 74:254–265.
Wolz R, Aljabar P, Hajnal JV, Hammers A, Rueckert D (2010a):
LEAP: Learning embeddings for atlas propagation. Neuro-
image 49:1316–1325.
Wolz R, Heckemann RA, Aljabar P, Hajnal JV, Hammers A,
Lotjonen J, Ruechert D & (2010b): Alzheimer’s Disease Neuro-
imaging Initiative. Neuroimage 49:2352–2365.
Wyman BT, Harvey DJ, Crawford K, Bernstein MA, Carmichael
O, Cole PE, Crane PK, DeCarli C, Fox NC, Gunter JL, Hill D,
Killiany RJ, Pachai C, Schwarz AJ, Schuff N, Senjem ML,
r
 r Sankar et al. r

Suhy J, Thompson PM, Weiner M, Jack CRJ (2013): Standardi-
zation of analysis sets for reporting results from ADNI MRI
data. Alzheimers Dement 9:332–337.
Yushkevich PA, Amaral RS, Augustinack JC, Bender AR,
Bernstein JD, Boccardi M, Bocchetta M, Burggren AC, Carr
VA, Chakravarty MM, Chetelat G, Daugherty AM, Davachi L,
Ding SL, Ekstrom A, Geerlings MI, Hassan A, Huang Y,
Iglesias JE, La Joie R, Kerchner GA, LaRocque KF, Libby LA,
Malykhin N, Mueller SG, Olsen RK, Palombo DJ, Parekh MB,
Pluta JB, Preston AR, Pruessner JC, Ranganath C, Raz N,
Schlichting ML, Schoemaker D, Singh S, Stark CE, Suthana N,
Tompary A, Turowski MM, Van Leemput K, Wagner AD,
Wang L, Winterburn JL, Wisse LE, Yassa MA, Zeineh MM
(2015): Quantitative comparison of 21 protocols for labeling
hippocampal subfields and parahippocampal subregions in in
vivo MRI: Towards a harmonized segmentation protocol.
Neuroimage 111:526–541.
Zuo XN, Anderson JS, Bellec P, Birn RM, Biswal BB, Blautzik J,
Breitner JC, Buckner RL, Calhoun VD, Castellanos FX, Chen
A, Chen B, Chen J, Chen X, Colcombe SJ, Courtney W,
Craddock RC, Di Martino A, Dong HM, Fu X, Gong Q,
Gorgolewski KJ, Han Y, He Y, He Y, Ho E, Holmes A, Hou
XH, Huckins J, Jiang T, Jiang Y, Kelley W, Kelly C, King M,
LaConte SM, Lainhart JE, Lei X, Li HJ, Li K, Li K, Lin Q,
Liu D, Liu J, Liu X, Liu Y, Lu G, Lu J, Luna B, Luo J,
Lurie D, Mao Y, Margulies DS, Mayer AR, Meindl T,
Meyerand ME, Nan W, Nielsen JA, O’Connor D, Paulsen D,
Prabhakaran V, Qi Z, Qiu J, Shao C, Shehzad Z, Tang W,
Villringer A, Wang H, Wang K, Wei D, Wei GX, Weng XC,
Wu X, Xu T, Yang N, Yang Z, Zang YF, Zhang L, Zhang Q,
Zhang Z, Zhang Z, Zhao K, Zhen Z, Zhou Y, Zhu XT,
Milham MP (2014): An open science resource for establishing
reliability and reproducibility in functional connectomics. Sci
Data 1:140049.

r 2896 r