See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/51783252

A Bayesian Framework for Automated

Cardiovascular Risk Scoring on Standard Lumbar
Radiographs

ARTICLE · NOVEMBER 2011

DOI: 10.1109/TMI.2011.2174646 · Source: PubMed

CITATIONS READS

3 29

7 AUTHORS, INCLUDING:

Peter Mysling Mads Nielsen

University of Copenhagen University of Copenhagen
7 PUBLICATIONS 9 CITATIONS 255 PUBLICATIONS 1,830 CITATIONS

SEE PROFILE SEE PROFILE

Lene Lillemark Sami Sebastian Brandt

3Shape University of Copenhagen
6 PUBLICATIONS 5 CITATIONS 70 PUBLICATIONS 814 CITATIONS

SEE PROFILE SEE PROFILE

Available from: Melanie Ganz

Retrieved on: 04 February 2016

A Bayesian Framework for
Automated Cardiovascular Risk Scoring on
Standard Lumbar Radiographs
Kersten Petersen, Melanie Ganz, Peter Mysling, Mads Nielsen,
Lene Lillemark, Alessandro Crimi, and Sami S. Brandt
Abstract—We present a fully automated framework for scoring
a patient’s risk of cardiovascular disease (CVD) and mortality
from a standard lateral radiograph of the lumbar aorta. The
framework segments abdominal aortic calcifications for com-
puting a CVD risk score and performs a survival analysis to
validate the score. Since the aorta is invisible on X-ray images,
its position is reasoned from (1) the shape and location of the
lumbar vertebrae and (2) the location, shape, and orientation
of potential calcifications. The proposed framework follows the
principle of Bayesian inference, which has several advantages
in the complex task of segmenting aortic calcifications. Bayesian
modeling allows us to compute CVD risk scores conditioned on
the seen calcifications by formulating distributions, dependencies,
and constraints on the unknown parameters. We evaluate the
framework on two datasets consisting of 351 and 462 standard
lumbar radiographs, respectively. Promising results indicate that
the framework has potential applications in diagnosis, treatment
planning, and the study of drug effects related to CVD.
Index Terms—Cardiovascular disease, risk scoring, radio-
graphs, Bayesian, segmentation, automated, calcifications, spine,
vertebrae, aorta, SMC sampler on shapes
Cardiovascular disease (CVD) is the most common cause
of death in the western world. In 2007, it accounted for
48% of all deaths in Europe [1], respectively 33.6% in the
United States [31]. According to pathological studies, the risk
for cardiovascular death positively and significantly correlates
with the evidence of atherosclerosis in the aorta and coronary
arteries [22]. Active and inflamed atherosclerotic plaques are
assumed to build up in the intimal walls of the arteries –
often unrecognized over many years – until they suddenly
provoke a heart attack or cardiac death. We therefore analyze
atherosclerotic plaques to develop effective indicators of CVD.
One important CVD risk factor are aortic calcifications in
the abdomen [33]. They correlate with coronary calcium levels
and are independent of the usual risk factors [34]. Further, they
have been linked to parental occurrence of premature CVD
[23] and vascular disease in patients with type II diabetes [26].
The severity of abdominal aortic calcifications is primar-
ily assessed with computerized tomography (CT) and X-ray
imaging. Although CT images are better suited for estimating
the degree of abdominal calcifications, standard X-ray images
Kersten Petersen, Melanie Ganz, Peter Mysling, Lene Lillemark, and
Alessandro Crimi are with the University of Copenhagen, DK-2100 Copen-
hagen OE, Denmark.
Mads Nielsen is with the University of Copenhagen, DK-2100 Copenhagen
OE, Denmark, and also with Nordic Bioscience Imaging and Synarc Imaging
Technologies, Herlev, Denmark.
Sami S. Brandt is with Nordic Bioscience Imaging and Synarc Imaging
Technologies, Herlev, Denmark.
1
have been used in epidemiological studies for identifying
CVD risk patients [21]. There are several reasons, why X-ray
imaging is beneficial for predicting cardiovascular mortality.
First, a lot of historical radiographs are available from large,
long lasting osteoporosis screening programs. Second, X-ray
machines are inexpensive and widespread, which is ideal for
conducting new, independent epidemiological studies. And
finally, the acquisition of X-ray images is fast and exposes
the patient to less radiation than CT.
However, it is not easy to delineate aortic calcifications in
lumbar standard radiographs. The images commonly have a
low signal-to-noise ratio, and the radiographic projection is
affected by the anatomy of the patient. In addition, one has to
account for different scanner settings, radiographer techniques,
patient positioning, and pathologies. Calcified plaques share
the same local appearance as the spine, bowel content, the
image rim, or artifacts in the radiograph, which explains,
why even trained radiologists struggle in segmenting aortic
calcifications accurately and reproducibly.
Since the manual computation of CVD risk scores is not
only subjective, but also laborious and expensive, the process
needs to be automated. The automatic segmentation of aortic
calcifications poses many challenges, though. To assess the
complexity of the problem, a random forest classifier [4]
was trained on calcifications labels and applied to a standard
radiograph. As illustrated in Fig. 1, the segmentation contains
lots of false positives, suggesting that context information is
needed to disambiguate the image content.
De Bruijne [5] proposed a technique that finds the aorta as
a region of interest before segmenting the aortic calcifications.
The main idea is to infer the aorta indirectly from the pose and
shape of the spine, and the relative location of potential aortic
calcifications. By modeling these dependencies, the scheme is
able to segment the aorta, even though the aorta boundary is
not visible on standard radiographs.
The approach was demonstrated with manually given land-
marks at the corners and midpoints of the first four lumbar
vertebrae. We can fully automate the scheme by generating the
vertebrae landmarks from an automated spine segmentation,
such as [10], [25], [30]. However, our experiments suggest that
the typical error of these automated vertebrae segmentation
methods suffices to mislead the aorta segmentation technique
of [5]. In several cases, this method places the aorta on top of
the vertebrae boundary or other false positive structures which
locally appear as calcifications [25].

(a)
Fig. 1. (a) Lateral standard radiograph of the lumbar aorta. (b) Manual annotations of the vertebrae L1-L4, the aorta, and aortic calcifications. (c) Output
from a random forest classifier based on a multiscale local jet representation of the calcifications and the background.
In this paper, we present a fully automated framework that
models all the steps from lumbar standard radiographs to the
CVD survival analysis. The framework reasons the aorta from
the segmented vertebrae and calcifications, but was shown
[25] to segment both the vertebrae and the aorta much more
accurately and reliably than [5]. Instead of concatenating two
segmentation techniques, we propose a unified model-based
system that is guided by the principles of Bayesian inference.
The Bayesian approach [27] uses the clear semantics of
probability theory to derive the best estimate, starting from a
requested set of properties. We formalize these properties by
probability distributions and dependency assumptions on the
parameters of our system. For instance, we explicitly express
our beliefs about likely vertebrae and aorta shapes, before
observing the image data. Similarly, we encode our prior
knowledge about the expected calcium distribution within the
aorta. We state, which parameters are assumed to be inde-
pendent of each other, and model conditional dependencies
and constraints of shape parameters [5], [24]. In Bayesian
theory, the observed image data is used in the likelihood
function for extracting additional information on the unknown
parameters. The combination of the prior model and the
likelihood function leads to the posterior distribution, which
captures the uncertainty in the unknown parameters, after
observing the image data. It represents the complete solution
to our problem and allows us to make predictions about likely
parameter settings and the associated scores, e.g., the CVD
risk score. The Bayesian paradigm thus provides a consistent
statistical interpretation of our domain knowledge and the
observed data. It should be seen in contrast to classical model-
based systems which propagate regularized point estimates.
The implementation of the framework contains several
refinements and novel ideas. The vertebrae segmentation,
for instance, relies on conditional shape models for steering
the search of likely vertebrae shapes. Another example are
2
(b) (c)
segment-based appearance, shape, and orientation features of
calcifications for improving the segmentation of the aorta.
The main contributions of this work, however, emerge on
a larger scale. 1) To the best of our knowledge, we present
the first feasible system for automatically computing the AC24
severity score [21] from a standard radiograph. 2) We provide a
complete Bayesian formulation of a challenging segmentation
problem where the correct solution is highly uncertain due
to a low signal-to-noise ratio, clutter, or occlusions. Our
framework naturally incorporates widely applicable ideas from
previous work, e.g., the static SMC sampler on shapes [25],
conditional point distribution models [24], or a spatial prior
for archipelago-like structures [15].
The remainder of this paper is organized as follows. In
Section I, we define the Bayesian framework and explain its
implementation. Section II shows experimental results. The
discussion of the results and the properties of the framework
are given in Section III.
I. M ETHOD
A. The Bayesian Framework
Let D = (DV , DC ) be the observed vertebrae and calcifica-
tion data, and may θA ∈ ΘA parameterize the unknown aorta.
Assume further that u ∈ YC N denotes a latent variable vector
for the N pixel locations, and that label space YC annotates
calcifications vs. background. Then the conditional expectation
of any score function f (u, θA ) is
Z
X
E {f |D} = f (u, θA )p(u, θA |D) dθA , (1)
ΘA
u∈YC N
P P P
where u∈YC N ≡ u1 ∈YC · · · uN ∈YC .
Assuming a Bayesian model, the joint posterior distribution
of the aorta parameters and calcification labelings given the
 3

Vertebrae Shape Model Aorta Shape Model Calc. Spatial Prior

p(θV ), Sec. I-B p(θA |DV ), Sec. I-D pspatial|location (u|θA ), Sec. I-F

Vertebrae Probability Map Aorta Probability Map Calc. Location Prior Compute Expected Score
p(v|θV ), Sec. I-C p(u|θA ), Sec. I-E plocation (u|θA ), Sec. I-F E {f |D}, Sec. I-A

Vertebrae Appearance Model Aorta Appearance Model Calc. Appearance Model

p(DV |v, θV ), Sec. I-C p(DA |u, θA ), Sec. I-E p(DC |, u, θA ), Sec. I-F

Vertebrae Posterior Aorta Posterior Calcification Posterior

p(θV |DV ), Sec. I-C p(θA |DV , DA ), Sec. I-E p(u|θA , DC ), Sec. I-F

Fig. 2. A model-based view of the Bayesian framework. The dashed boxes denote posterior distributions that depend on all the models they encompass. This
includes appearance models for representing likelihoods, and models describing prior knowledge about the shapes and spatial distribution of the vertebrae,
the aorta, and aortic calcifications. The score is computed for samples from the joint posterior distribution of aorta shape parameters θA and calcification
labelings u given the observed data D.

data, p(u, θA |D), is the complete solution of our segmentation that the aorta appearance model, p(DA |u, θA ), and the aorta
problem. It factorizes into probability map, p(u|, θA ) are independent of DV (see Section
I-E). We obtain DA by denoising and classifying DC , which
p(u, θA |D) = p(u|θA , DC )p(θA |D) . (2)
shall simulate the behavior of the more costly calcification
We model the likelihood, p(u|θA , DC ), by a latent variable spatial prior.
model [18] of calcifications, and assume independence of The aorta parameters θA are related to the vertebrae pa-
the vertebrae data, DV . Applying Bayes’ theorem, the latent rameters θV ∈ ΘV , which serve as nuisance parameters in our
variable model model. Thus, the posterior distribution of the aorta parameters,
given the vertebrae data, in (5) can be written as
p(u|θA , DC ) ∝ p(DC |u, θA )p(u|θA ) (3) Z
gives rise to a likelihood function, p(DC |u, θA ), and a prior, p(θA |DV ) = p(θV , θA |DV ) dθV
p(u|θA ). The likelihood is given by a calcification appearance ZΘV
model, while the prior = p(θA |θV )p(θV |DV ) dθV . (6)
ΘV
p(u|θA ) = plocation (u|θA )pspatial|location (u|θA ) (4)
It is referred to as the aorta shape model. The conditional
is the product of a calcification location prior and a calcifi- model p(θA |θV ) regresses aorta from vertebrae parameters,
cation spatial prior (see Section I-F). and is considered to be independent of DV (see Section I-D).
According to [25], we can rewrite Analogous to (5), the posterior distribution of the vertebrae
parameters, given the vertebrae data, unfolds as
p(θA |D) ≈ p(θA |DV , DA )
X NV X
Y
∝ p(DA |u, θA , DV )p(u, θA |DV ) p(θV |DV ) ∝ p(θV ) p(yn |vn , θV )p(vn |θV ) , (7)
u∈YC NA n=1 vn ∈YV
X
= p(θA |DV ) p(DA |u, θA )p(u|θA )
where DV = {yn }N n=1 represents the vertebrae data and v =
V
u∈YC NA NV
{vn }n=1 the latent variable for the vertebrae labels. We obtain
X NA
Y a vertebrae appearance model, a vertebrae probability map,
= p(θA |DV ) p(xn |un , θA )p(un |θA )
and a vertebrae shape model (see Section I-B and I-I).
u∈YC NA n=1
Fig. 2 provides an overview of our framework. The graph
NA X
Y indicates that the vertebrae, aorta, and calcification posterior
= p(θA |DV ) p(xn |un , θA )p(un |θA ) ,
distribution each consist of an appearance model, and two
n=1 un ∈YC
(5) priors. The shape models and the spatial prior introduce prior
knowledge about the shape of an object, whereas the class
where DA = {xn }N A
n=1 encodes the appearance, shape, and probability maps and the location prior encode the expected
orientation of calcification segments and where we assume segmentation result prior to observing the data.

B. Vertebrae Shape Model
We capture the statistics of vertebrae shapes with the
linear Point Distribution Model (PDM) proposed by Cootes
et al. [9]. The model learns possible variations in shape
from a training set and yields both an efficient and com-
pact parameterization. The PDM requires that each shape
x = (x1 , . . . , xN , y1 , . . . , yN )T is described by N landmarks
at geometrically corresponding positions over the training set
and distinguishes two essential learning steps.
To establish a geometrical frame of reference, the training
shapes are first aligned to a mean shape by translating, rotating
and scaling them, so as to minimize the sum of squared
distances between the landmarks. This method is known as
Generalized Procrustes Analysis (GPA). Formally, this simi-
larity transformation maps the shape x to the normalized shape
x̃ in Procrustes space using the pose parameters
a = (s, α, tx , ty )T , (8)
where s denotes the scale, α the rotation angle, and (tx , ty )
the translation.
Second, the variability of the aligned shapes is modeled
by a Principal Component Analysis (PCA). The Procrustes
space coordinates x̃ are projected onto the first D principal
components, where D is chosen such that a certain proportion,
fv , of the shape variation in the training set is explained.
The reduced set of principal components lowers the risk of
overfitting and improves speed and compactness of the model.
The projection coefficients are called shape parameters
˜) ,
b = P(x̃ − x̄ (9)
where x̄ ˜ is the mean in Procrustes space and P =
(p1 , . . . , pD ) refers to the orthonormal eigenvectors that cor-
respondto the D largest eigenvalues of covariance matrix Σx̃ .
In the formulation of the PDM, it is assumed that pose and
shape are normally distributed, but independent of each other.
Let the mean of the combined pose and shape parameters be
!
ā
θ̄ = , (10)
b̄
where ā and b̄ denote the means of pose and shape over the
training set, respectively.
For generalizing to more testing shapes, the diagonal
Tikhonov regularizer Γa is added to the pose part of the
covariance matrix. The regularized covariance matrix is
!
A + Γa 0
Σ= , (11)
0 B
with A and B as the covariance matrices for pose and shape.
The vertebrae shape distribution θV is then given by


p(θV ) = N θ̄V , ΣV , (12)
where N denotes the Gaussian probability density function
with mean θ̄V and covariance ΣV . We refer to it as the
vertebrae shape model.
Note that the simple form of the normal distribution facil-
itates the application of the trained PDM. It allows to easily
generate new shape samples, as well as to assign probabilities
to shape examples.
4
C. Vertebrae Posterior
The vertebrae posterior distribution in (7) consists of three
models: the vertebrae shape model (Section I-B), p(θV ), which
is the prior model (12) for the vertebrae shapes; the vertebrae
probability map, p(v|θV ), which encodes the expected spatial
distribution of vertebrae structures; and the vertebrae appear-
ance model, p(DV |v, θV ), corresponding to the likelihood. The
latter two models are detailed in the following.
The vertebrae appearance model is represented by a random
forest classifier [4] that models the likelihood of the vertebrae
data. The training images are described by a set of local image
features, e.g., the multiscale local jet [14]. The associated
labels are manually provided by an expert (see Fig. 3(a)).
As in [25], the labeling map consists of six classes: anterior
background, posterior background, inter vertebrae space, ver-
tebrae boundary, intra vertebrae space, and background. We
also include a rejection region (black region in Fig. 3(a))
between the vertebrae boundary and the inter vertebrae space.
The vertebrae probability map, p(v|θV ), reflects the ex-
pected probability that the n-th pixel in the vertebrae re-
gion belongs to the k-th label region with k ∈ 1, . . . , K.
We construct the map by applying the vertebrae appearance
model with the learned labeling to a second set of validation
images. The obtained probabilistic outputs of the random
forest classifier are averaged to form the vertebrae probability
map. The probability maps of the vertebrae boundary class
for the manual labeling map are depicted in Fig. 3(b). Our
experiments indicate that the probabilistic label assignments
lead to slightly higher accuracy and robustness than hard label
assignments, which are used in [13], [25], and [6].
(a) (b)
Fig. 3. (a) The manual labeling map for the vertebrae. (b) The class
probability map of the vertebrae boundary region of the manual labeling map.
D. Aorta Shape Model
We also build a PDM for modeling the probability distribu-
tion over aorta shapes. However, the parameter training differs,
as the aorta depends on the vertebrae in pose and shape. We
follow the approach by de Bruijne [5] in using the same pose
parameters for the aorta as for the vertebrae, and modelling
the shape parameters by a conditional normal distribution.
 5

In detail, aorta shapes are aligned with the same pose the same local appearance as other structures in the image,
parameters as the corresponding vertebrae shapes to compare e.g., image artifacts, bowel content, the vertebrae boundary
the correlation between aorta and vertebrae shapes, i.e., or the image rim. Thus, more global information have to be
considered to rule out these false positives.
āA|V = āV (13)
and
AA|V = AV . (14)
As soon as the aorta and vertebrae shapes have been trans-
formed by GPA, the shape parameters of the aorta are re-
gressed from the vertebrae using a conditional normal distri-
bution. The mean conditioned on the vertebrae is given by
b̄A|V = b̄A + BAV B−1
V bV − b̄V , (15)
and the covariance matrix is computed as
BA|V = BA − BAV B−1
V BVA , (16)
where the covariance matrices are obtained from the joint
covariance matrix of bA and bV
!
BA BAV
B= . (17)
BVA BV
The conditional covariance matrix BA|V is the Schur com-
plement of BV in B, i.e., the inverse of the inverted joint (a) (b)
covariance matrix, where the vertebrae part has been dropped. Fig. 5. (a) The output from a statistical pixel classifier in a region of interest
By assuming a Gaussian model, with the mean and covari- for the aorta. (b) The denoised output, where vertebrae, the image rim, and
ances above, the probability of an aorta shape θA conditioned small calcifications have been removed for the sake of robustness.
on a vertebrae shape θV is given by

The calcification probabilities are denoised as follows.
p(θA |θV ) = N θ̄A|V , ΣA|V , (18) Pixels that lie on the image rim or appear in at least half
of the sampled vertebrae region are set to c. Morphological
where !
āA|V filters (opening, followed by closing) clean segments that are
θ̄A|V = (19) smaller than the smallest manually annotated calcification in
b̄A|V the training data. Weakly connected components are separated
and ! by morphological opening. The probabilities of all modified
AA|V + Γa 0 pixel classifier outputs are set to c. Fig. 5(b) shows that only
ΣA|V = . (20) segments remain that are larger than a detectable calcification.
0 BA|V
Segments that have the size of big calcifications are ana-
Thus, the aorta shape model is specified by substituting (18) lyzed with respect to their appearance, shape, and orientation.
and (7) into (6). Smaller segments, for which these features cannot be robustly
evaluated, are assumed equally likely to be calcified or not.
E. Aorta Posterior We apply a segment-based random forest classifier to
evaluate the large segments. The input data characterizes
The posterior distribution of the aorta samples in (5) factor-
the appearance and shape of the segments, as well as their
izes similar to the vertebrae posterior distribution. The aorta
orientation with respect to the aorta. More specifically, we
shape model, p(θA |θV ), has been defined in (6); the aorta
use the mean and median of the segment intensities, the
probability map, p(u|θA ), describes the spatial distribution
area, convex area, diameter, perimeter, length, width, and
of aorta-related structures; and the aorta appearance model,
the angle between the major principal component and the
p(DA |u, θA ), represents the likelihood. Let us treat the latter
nearest aorta wall. The training labels are gained from the
two models in more depth.
manual calcification annotations. The probabilistic output of
The aorta appearance model is composed from the denoised
the segment classifier (see Fig. 4(b) and Fig. 4(d)) combined
output of a random forest classifier for calcifications, i.e.,
( with the fixed probabilities for the small segments constitute
c noise at position n the likelihood p(DA |u, θA ).
p(xn |un , θA ) = , (21) The aorta probability map is constructed from the manual
p(xn |un , θA ) otherwise
annotations of calcifications to describe the spatial distribution
where c > 0 is a small constant. The denoising step is of a calcification and the background class. We compute
necessary, because the pixel classifier alone cannot reliably the expected probability distribution of calcium given the
segment calcifications. The problem is that calcifications share mean aorta shape by following the approach in [5]. A mean

(a) (b)
Fig. 4. (a) Comparison of a misplaced aorta sample (dashed) and the manually annotated aorta (solid). (b) Segment probabilities of the misplaced aorta
sample. (c) Comparison of a well fitting aorta sample and the manually annotated aorta. (d) Segment probabilities of the well fitting aorta sample. (e) The
aorta probability map for the calcification class.
label vector is created by averaging the calcification labels
of an aligned training set (see Fig. 4(e)). The regularized
version of this mean label vector defines the distribution of the
calcified class, while the complementary probabilities estimate
the distribution of the background class.
F. Calcification Posterior
The latent variable model, p(u|θA , DC ), in (3) models the
posterior distribution of the calcification labels and it factorizes
into a likelihood and a prior model.
The likelihood, p(DC |u, θA ), is called the calcification
appearance model and relies on the pixel classifier of the
aorta appearance model. False positives are suppressed, as
described in Section I-E, but in contrast to the aorta appearance
model no segment classifier is applied to evaluate the segment
probabilities. Instead, the calcification spatial prior, arising
from (4), models the spatial composition of the calcifications.
(a) (b) (c)
Fig. 6. (a) Manual annotation of calcifications. (b) Output probabilities of
calcification pixel classifier. (c) Conditional mean of the calcification posterior
for samples from the calcification location prior.
The calcification spatial prior, pspatial|location (u|θA ), is con-
structed from a binary patch dictionary using a patch grammar
based on Markov mesh random fields. [15], as summarized
below. The effect of the prior is illustrated in Fig. 6.
In the dictionary training phase, S binary patches of size
2
M × M , X ∈ {0, 1}S×M , are approximated by a linear
6
(c) (d) (e)
combination of patch prototypes. The goal is to minimize
E = ||X − DA||2fro , (22)
2
M ×K
where D ∈ {0, 1} is a dictionary of K binary patches,
and A ∈ {0, 1}K×S is a sparse representation of X with
respect to D, such that each column of A contains one
non-zero entry. Instead of finding the global minimizer of
(22), the solution is approximated by first learning D with
K-means clustering for a set of training patches estimated
from manual annotations of the calcifications. After computing
D, the estimate for A is determined by selecting the closest
prototype for each training patch.
To account for correlations among neighboring patches,
their joint distribution is modeled by a first-order Markov
Mesh Random Field (MMRF) [17]. Using the chain rule of
probability and the Markov assumption, we obtain
pspatial|location (u|θA ) = p(u1 )p(u2 |u1 ) . . . p(uT |u1 , . . . uT −1 )
T
Y
∼
= p(ut |N(u)) , (23)
t=1
where N(u) comprises all the direct neighbors of patch u
that have previously been processed, and T is the number of
patches in the image. The conditional distributions in (23) are
estimated from the histogram of patch neighbor combinations
in the training data. The parameters, M and K, are selected
via the minimum description length (MDL) principle.
The calcification location prior, plocation (u|θA ), is built in
the same way as the probability map for the calcification label
in the aorta probability map (see Section I-E). It encodes
our prior knowledge about the aorta locations, at which
calcifications are likely to appear.
G. Sampling of the Posterior
The Bayesian framework contains four intractable distri-
butions, which we approximate with Monte Carlo simula-
tion [12]. The essence is to generate a large collection of
 7

(a) (b) (c) (d)

Fig. 7. (a) Initial vertebrae samples. (b) Output samples from vertebrae SMC sampler (black) and manually annotated vertebrae (white dashed). (c) Initial
aorta samples. (d) Output samples from aorta SMC sampler.

θ̄V , ΣV θ̄A , ΣA , ΣAV , ΣVA

Vertebrae Aorta
Shape Model Shape Model

θV θA θA

Calcification
Location Prior
f
D
Vertebrae Aorta
Posterior Posterior
u
Calcification
LV LA Spatial Prior
MC LC
Calcification
Pixel Classifier

DV DA DC

| {z } | {z } | {z }
Vertebrae Aorta Calcifications

Fig. 8. Representation of the Bayesian framework as a probabilistic graphical model using plate notation [7] with factor graphs. Nodes with a single circle
refer to probabilistic variables that are either shaded, if they are observable, or empty, if they are latent. Nodes with double circles are deterministic functions
of their inputs, e.g., the score function f . Factor nodes describe models, while small black circles indicate model parameters. Boxes that enclose variables
are called plates. They group variables into a sub-graph which is replicated as often as it is specified in the bottom right corner. Edges that cross a plate
boundary are repeated once, whereas touching edges are replicated as often as the subgraph. We use them to point out the Monte Carlo estimates of the
posterior distributions in the vertebrae, the aorta, and the calcification stage.

L weighted random samples {θ(l) , w(l) }L

l=1 , such that their We employ the static SMC sampler on shapes [25] to obtain
empirical distribution the Monte Carlo estimate of shape samples (see Section I-H).
L Based on the static SMC sampler [11] the method sequentially
X
p̂(θ) = w(l) δ(θ − θ(l) ) , (24) samples from a sequence of target distributions known up to
l=1 a normalizing constant and defined on a common space.
Consequently, the vertebrae SMC sampler (see Section
where δ denotes the Kronecker delta, asymptotically converges
I-I) produces a weighted ensemble of vertebrae samples
to the target distribution p(θ). That is, for any p-integrable
{(θV , wV )}L V
l=1 that approximates the vertebrae posterior
function f : Θ → R it holds that
p(θV |DV ) ≈ p̂(θV |DV )
L
X Z
(l) (l) L→∞ LV
w f (θ ) −→ f (θ)p(θ) dθ , (25) X (l)

(l)

a.s. Θ = wV δ θV − θV . (26)
l=1
l=1
where a.s. denotes almost sure convergence.
 8

Similarly, the posterior distribution of the aorta, are defined relative to the mean shape and warped to each
p(θA |DV , DA ), in (5) is approximated by the aorta SMC shape sample using thin plate splines [25], so as to statistically
sampler. It uses the Monte Carlo estimate for the posterior of compare shapes of different sizes. The term p(θ) is interpreted
the vertebrae, p(θV |DV ), and generates a weighted ensemble as the probability of θ under a shape model prior to seeing the
of aorta samples {(θA , wA )}L A
l=1 , such that data; the class probability map p(un |θ) is a second prior that
encodes the expected spatial distribution of contextual struc-
p(θA |DV , DA ) ≈ p̂(θA |DV , DA ) tures; and the likelihood, p(xn |un , θ), describes a statistical
LA
X (l)

(l)
 pixel classifier that models non-linear appearance information
= wA δ θA − θA . (27) at the n-th location.
l=1
The static SMC sampler is similar to particle filtering,
Finally, evaluating the expected quantities in (1), i.e., but with an important difference. A sequence of artificial
−1
target distributions {πt }Tt=1 is introduced, so as to smoothly
E {f |D}
Z guide the tractable proposal distribution π1 = p(θ) to the
X
= f (u, θA )p(u, θA |D) dθA target distribution πT = π(θ|D). In other words, the static
ΘA
u∈YC N
SMC sampler gradually propagates samples to regions of
(2)
Z X high density instead of sampling from the complicated target
= f (u, θA )p(u|θA , DC )p(θA |D) dθA distribution π(θ|D) directly.
ΘA N
u∈YC
  The artificial target distributions are defined by the geomet-
Z X ric path between the proposal and the target distribution, or
= p(θA |D)  f (u, θA )p(u|θA , DC ) dθA ,
ΘA
u∈YC N πt (θ) = π(θ|D)βt p(θ)1−βt , (31)
(28)
where the annealing variables 0 ≤ β1 < . . . < βT = 1
leads to two nested Monte Carlo simulations. By using (5), determine the complexity of the t-th target distribution. The
LC aorta samples are drawn from (27), after which MC annealing parameters are either fixed in a cooling schedule,
(l)
latent variable samples u(m,l) ∼ p(u|θA , DC ) are generated or, as for our application, iteratively optimized to achieve a
conditioned on each aorta sample. This yields pre-defined number of effective samples.
LC X
MC The static SMC sampler is a sequential importance re-
1 X (l) sampling scheme, where each artificial target distribution
E {f |D} ≈ f (u(m,l) , θA ) . (29) (l) (l)
LC MC m=1 πt (θ) is approximated by L weighted samples, {θt , wt }L l=1 .
l=1
For each iteration, the algorithm passes through a sampling,
Fig. 8 illustrates the conditional dependencies between
weighting, and resampling phase. First, a collection of sam-
observed and latent variables, and also shows the interplay (l)
ples, {θt }L l=1 , is drawn, either from p(θ) (when t = 1), or
of Monte Carlo estimates, models, and key parameters.
from the output samples of the previous iteration (when t > 1).
Second, the samples are weighted according to the relative
H. Static SMC Sampler on Shapes posterior densities. Assuming that two subsequent artificial
The static SMC sampler on shapes [25] is used to sample target distributions are close to each other, i.e., πt ≈ πt+1 ,
vertebrae and aorta shape distributions (see also Section I-I). It the unnormalized importance weights are derived [11] as
usually performs better than MCMC methods, which can eas- πt (θt−1 )
ily get trapped in local modes and allows to efficiently explore wt ∝ wt−1 . (32)
πt−1 (θt−1 )
high-dimensional static spaces, while providing asymptotically
consistent estimates of the target distribution. The method is Finally, the samples are resampled according to the normalized
suited for segmentation problems, where the correct solution weights. This means that samples from high-density regions
is highly uncertain due to, for instance, a low signal-to-noise generate perturbed copies of themselves, whereas unlikely
ratio, clutter, or occlusions. It is able to provide multiple samples die out.
hypotheses and can incorporate prior knowledge about an The conditional probability p(θt+1 |θt ) from sample θt to a
object’s shape or contextual structures. modified version θt+1 is called the forward Markov kernel of
The goal is to draw samples from the posterior distribution the static SMC sampler. For t = 1, a large number of samples,
of shapes θ given the observed data D, i.e. (see 5), {θ(l) }L
l=1 with Ldense ≫ L, is drawn from p(θ) to construct
dense

N X local PDMs. A local PDM is trained from the Mt nearest

Y
p(θ|D) ∝ p(θ) p(xn |un , θ)p(un |θ) neighbors of a given shape sample θt and used to produce
n=1 un ∈Y new shapes θt+1 that are plausible under the global shape
model p(θ). Initially, Mt is large to ensure a good mixture of
≡ π(θ|D) , (30)
samples; later it is decreased to reduce the weight variance.
where the data D = {xn }N n=1 is assumed to be i.i.d. and The static SMC sampler on shapes terminates, as soon as
u = (u1 , . . . , uN )T ∈ Y N denotes a latent variable vector; Y βt = 1, or a pre-defined number of sampling iterations has
defines the space of class labels, and N stands for the number been reached. Fig. 7 illustrates how far the initial vertebrae
of evaluation points for each shape. The evaluation points and aorta samples have evolved when the algorithm stops.

I. Vertebrae SMC Sampler
The vertebrae SMC sampler extends the static SMC sampler
on shapes in two ways. First, NV /K evaluation points are
sampled from each vertebrae label class, since the classes are
assumed to be roughly equally important for the vertebrae seg-
mentation. The background region, however, is not sampled,
as it is hardly discriminative for a given vertebrae sample.
And second, we introduce a modified version of the forward
Markov kernel. In the standard setting (see Section I-H), new
samples are drawn from a local PDM with respect to a source
sample. We refine this model by adding several conditional
PDMs (see Section I-D), that conditioned on a random subset
of vertebrae predict the residual vertebrae. An example is
shown in Fig. 9(a), where we sample the vertebrae L1 and L4
depending on the fixed vertebrae L2 and L3. The underlying
idea is to guess the landmarks that fit well to the data and
only perturb the remaining, possibly misplaced landmarks.
(a) (b)
Fig. 9. (a) A conditional model from a source sample (solid) to L1 and
L4 (dashed). (b) A conditional model for generating a shifted L1 vertebrae.
Vertebrae L1 to L3 from the source sample are assumed to be L2 to L4 of
the new sample.
The conditional PDMs may not only predict vertebrae on
the same vertebral level as the source sample, but also on
adjacent vertebral levels. In Fig. 9(b), L1 of the new sample
is inferred from L2-L4, which corresponds to L1-L3 in the
source sample. This may be beneficial when the mode of the
target distribution is peaked, and none of the initial vertebrae
samples fits well enough to survive the filtering process. It
allows to create samples based on sub-parts of surviving, but
shifted vertebrae samples.
The local PDM and the conditional PDMs are randomly
chosen for resampling. Most of the initial samples stem from
the local PDM, whereas resampled shapes are drawn with
increasing probability from the conditional PDM.
II. E XPERIMENTS
A. Data Collection
The Bayesian framework is evaluated on lateral standard
radiographs of the lumbar aorta and vertebrae. The radio-
graphs were acquired in 1992 (baseline) and 2001 (follow-
up) within a combined osteoporosis-atherosclerosis screening
9
PMA , LPC , and SPC
SMV , SMA , PCV , PCA , and PCC
SCA
LMV PMV
| {z } | {z }
Training data Test data
Fig. 10. The data partitions for training and testing the models of the
Bayesian framework in one fold of the cross-validation. The abbreviations
in the table are as follows. PM: probability map, LP: location prior, SP:
spatial prior, SM: shape model, PC: pixel classifier, SC: segment classifier,
LM: labeling map; V: vertebrae, A: aorta, and C: calcifications.
program (EPIPF sub-study) [2]. Later they were digitized
using a Vidar Dosimetry Pro Advantage scanner at 570 dpi
producing 9651×4008 pixels on a 12-bit gray scale. Out of
the available images, we selected those, where the lumbar
vertebrae L1-L4 and the corresponding aorta segment are
entirely included in the image. This leaves us with 351 usable
baseline, respectively 463 follow-up images.
Trained, blinded radiologists annotated the scanned images
on a Sectra radiological reading unit. The radiologists were
instructed to place six landmarks at the corners and end-
plate mid-points of the vertebrae L1-L4, as it is commonly
performed for vertebral height measurements [16]. In addition,
they were asked to delineate the aorta walls and each visible
aortic calcification in the lumbar region.
The radiographs vary considerably in acquisition quality
and visible pathologies. Especially, the baseline radiographs
are impaired by image artifacts, contrast changes, occluding
objects, and different patient positioning. Also, the subject’s
biology is diverse. Healthy and fractured vertebrae are present,
as well as no to many calcifications.
B. Model Evaluation
We perform 5-fold cross-validation to estimate the general-
ization ability of the Bayesian framework. In each round of
cross-validation, the available training data is split to prevent
overfitting of the individual models. However, as the data is
limited, the data cannot be disjointly distributed among the
models. Instead, we allow models to be trained on the same
data if they are at most weakly dependent on each other. Fig.
10 shows our proposed splitting of the data for one fold.
The calcification annotations of the entire training set are
used for training the aorta probability map, the calcification
location prior, and the calcification spatial prior. We assume
that these models have only indirect dependencies among each
other and the remaining models of the framework.
However, 1/5 of the available training data is reserved for
creating the segment classifier within the aorta appearance
model. When learning informative appearance, shape, and
orientation features for the segments, it is important to be
independent of the proposed aorta and vertebrae shapes, as
well as the statistical pixel classifiers which are involved in
constructing the segments.
The remaining 4/5 of the training data are used for training
the residual models. The vertebrae shape model and the
aorta shape model rely on different annotations, while the
statistical pixel classifiers for the vertebrae, the aorta, and

the calcifications use the image data. We assume that the
dependencies among these models are negligible, and that their
influence on the vertebrae maps is small as well. The vertebrae
probability map and the vertebrae labeling map are trained
on two equally sized independent datasets, since the statistical
pixel classifier for constructing the probability map should be
trained on a separate dataset.
C. Parameter Settings
1) Shape Models and Spatial Prior: The shape models for
both the vertebrae and the aorta explain fv = 0.95 of the shape
variation, which corresponds to 16 principal components for
the vertebrae, respectively 8 for the aorta. Both models are
regularized by a small value Γa = diag(1, 1, 1, 1) × 10−4 .
As the number of aorta landmarks varies, the aorta shape is
represented by 21 linearly interpolated points from each aorta
wall. The parameters for the calcification spatial prior are set
according to [15].
2) Class Probability Maps: The region for the vertebrae
probability map is obtained by enlarging the mean vertebrae
shape by 9 mm in each direction. The safety margin around
this vertebrae region is 12 mm wide. In this corridor, pixels are
neither assigned to a vertebrae region nor to the background.
The number of vertebrae region clusters for the automated
labeling map, K, was set to 18 after cross validating all values
between 5 and 25.
3) Appearance Models: The three statistical pixel classi-
fiers and the segment classifier are modeled by random forest
classifiers with 200 decision trees and 7 random split variables.
These values have been determined by cross validation on a
separate dataset. The appearance features for the pixel clas-
sifiers include the original image and the multiscale local jet
up to the third order at three logarithmically increasing scales
(0.18 mm, 0.56 mm, and 1.78 mm). Of the available training
data, 105 observations are randomly sampled to generate the
input data of the respective classifier. The training data of the
vertebrae pixel classifier contains equally many samples from
the vertebrae classes, while the aorta and calcification pixel
classifier are trained on 20% calcification and 80% background
pixels. The classifier output probabilities are thresholded at
0.85 for maximizing the calcification area overlap. The value
has been determined on an independent validation set.
4) Sampling Methods: The static SMC samplers for the
vertebrae and the aorta use the same parameter settings as
in [25]. The posteriors are estimated by LV = 300 and
LA = 100 samples in the final iteration. For the calcification
posterior, we set LC = 20 and MC = 10. We sample from
all possible conditional models for the vertebrae, where the
conditioned variables include L2 or L3, the vertebrae with the
least uncertainty (see Fig. 7(b)). The ratio of the conditional
models and the local shape model is linearly increased from
0 to 0.5 during the sampling process. This has the effect
that in the initial exploration phase of the sampler, nearly all
the samples are drawn from the local PDM, whereas with
increasing certainty in the best fit, conditional PDMs are used
to refine the search.
10
TABLE I
P ROPORTION OF SUCCESSFUL , LEVEL - SHIFTED AND FAILED
VERTEBRAE SEGMENTATIONS
Dataset Successful Level-shifted Failures
Baseline 270 (77%) 74 (21%) 7 (2%)
Follow-up 332 (72%) 126 (27%) 5 (1%)
TABLE II
P OINT- TO - CONTOUR ERROR STATISTICS OF THE AUTOMATED
FRAMEWORK FOR SUCCESSFUL CASES
Dataset Mean Median 75%-ile 95%-ile Errors > 2 mm
(mm) (mm) (mm) (mm)
Baseline 1.22 1.09 1.32 2.22 4.2%
Follow-up 1.34 1.24 1.49 2.45 6.3%
D. Results
We evaluate the framework on the baseline (bl) and the
follow-up (fu) dataset, both fully automated (Auto) and
conditioned on one manually placed spine landmark (Semi).
The semi-automated setup is meant to estimate the influence
of vertebral shifts, the most common error of the vertebrae
segmentation. In accordance with previous work, we present
the segmentation results for the vertebrae, the aorta, and the
calcifications only for the non-level-shifted cases. For our
framework, this restriction is indicated by a superscript plus
sign (Auto+ ). The computation of the CVD risk scores and
the survival analysis, however, is based on Auto and Semi,
the two setups that process the entire dataset.
It takes approximately 50 minutes on a 2.4 GHz processor
with 4 GB RAM to automatically compute the CVD risk score
for a given radiograph. The total training and testing time of
all our experiments (≈ 1600 images) amounts to roughly two
days on a server with 60 nodes, even though our code is written
in MATLAB and not optimized for speed. In comparison, a
trained radiologist requires nearly an hour for delineating the
aortic calcifications of one standard radiograph.
1) Vertebrae segmentation: On standard radiographs, it is
often difficult to determine the correct height of the lumbar
vertebrae (see Fig. 11(i)). Even trained radiologists may con-
fuse the vertebral levels if they cannot rely on complimentary
vertebrae scans. To assess the robustness of our system, we
have categorized the vertebrae segmentation results in Table
I, following the definitions of [30]. The possible classes are
successful, level-shifted (by one vertebral level), and failure.
Table II summarizes the point-to-contour error statistics
between the automatically and manually segmented vertebrae
for the successful cases. For each dataset, the table presents the
mean, median, 75th and 95th percentiles; and the proportions
of point errors larger than 2 mm.
Table III shows that our technique is competitive with previ-
ous work on vertebrae segmentation for standard radiographs.
The mean point-to-contour error of Auto+ compares to the
state-of-the art, although our method is trained on only six
landmarks per vertebra. The work of [19] suggests that our
method will be more accurate if more vertebrae landmarks are
available. The robustness of our method is at least as good as
[30], considering that Roberts et al. have cropped the images
 11

TABLE III TABLE V

S UMMARY OF VERTEBRAE SEGMENTATIONS INCLUDING MEAN M EAN JACCARD INDEX J¯ WITH STANDARD DEVIATION FOR
POINT- TO - CONTOUR ERROR CALCIFICATION SEGMENTATION

Authors Modality Shifted Failure Mean Authors J¯ std(J)

(mm) This study (Auto+
bl ) 0.18 0.16
Semi-automated Techniques This study (Auto+
fu ) 0.28 0.16
Roberts et al. [29] DXA 0% 0% 0.80
Roberts et al. [28] X-ray 0% 0% 0.65
7
Iglesias et al. [19] X-ray 0% 0% 0.50 Radiologist
Semi−automatic
Automated Techniques 6 Fully automatic
de Bruijne et al. [6] X-ray N/A N/A 1.40
5
Roberts et al. [30] X-ray 20% 7% 1.06

Hazard ratio
This study (Auto+
bl ) X-ray 21% 2% 1.22 4

This study (Auto+

fu ) X-ray 27% 1% 1.34 3

TABLE IV 2
P OINT- TO - CONTOUR ERROR STATISTICS AND MEAN JACCARD INDEX J¯
FOR AORTA SEGMENTATION 1

0
Authors Mean Median 75%-ile 95%-ile J¯ 0 1 2 3 4 5
Time (years)
(mm) (mm) (mm) (mm)
Semi-automated Techniques Fig. 12. Hazard ratios for the EPIPF baseline study showing the relative CVD
de Bruijne [5] 2.70 N/A N/A N/A 77% death risk for patients in the high-risk group (AC24 score > 3.5) compared
to patients in the low-risk group (AC24 score ≤ 3.5).
Automated Techniques
This study (Auto+
bl ) 2.61 2.33 3.32 5.05 73%
This study (Auto+
fu ) 2.62 2.21 3.18 5.01 73%
As a reference, the mean Jaccard index between two trained
radiologists was 0.51 on the follow-up dataset. Note that the
below L4 to reduce the number of ambiguities. Our technique Jaccard index is a strict error measure that is very sensitive
shifts by one vertebral level up- or downwards, as often as to disagreements on small areas, as it is often the case in the
[30] shifts solely upwards. Moreover, our method fails for baseline dataset (see Fig. 11(g)). It is difficult to compare our
considerably fewer cases, and except for two baseline cases results with previous work, since in [5] only the accuracy and
all these failures are shifts by two vertebral levels. Cohen’s κ coefficient [8] were evaluated. Both of these error
2) Aorta segmentation: In previous work [25], we demon- measures, however, are problematic, since they are dominated
strated that our aorta segmentation method clearly outperforms by the large number of true negatives.
the concatenation of [6] and [5]. On a followup sub-study, 4) Computation of CVD risk score: Although the area
a preliminary version of our system achieved 71% for the overlap between the automatically and manually computed
median Jaccard index [20], median(J), calcifications is relatively small, the results are useful for
quantifying the severity of aortic calcifications using the
|A ∩ B| popular AC24 antero-posterior severity score [21]. The idea
J(A, B) = , (33)
|A ∪ B| of the AC24 score is to assess the location and severity of
whereas the automation using the concatenated approach got calcifications by a composite score between 0 and 24. For
51%. In comparison, the mean Jaccard index, J, ¯ between two this score, the aorta is divided into four segments parallel to
trained radiologists was 80% on 29 available annotations, re- the vertebrae L1-L4, and the severity of anterior and posterior
spectively 87% on 21 annotations among the same radiologist. calcifications in each segment is graded separately on a 0-3
In this work, we compare our automated approach to a semi- scale. The correlation coefficient between the automated and
automated technique [5] (see Table IV). We report the same manual AC24 scows is r = 0.7.
statistics as for the vertebrae segmentation but exclude the 5) Survival analysis: For the survival analysis, we divided
number of errors larger than 2 mm. Instead we present the the CVD patients into two approximately equally sized groups
mean Jaccard index J. ¯ The statistics show that our automated by thresholding the AC24 score at 3.5. For each group, i =
approach performs similar to [5], where we have converted 1, 2, we separately estimated the probability density p(t) and
the Dice score D into the Jaccard index J by computing the probability distribution P (t) = P (T < t) of the survival
D time T by employing a Gaussian kernel density estimator with
J= . (34) automatic bandwidth selection [3]. Using the kernel estimates,
2−D
we computed the hazard functions
3) Calcification segmentation: Fig. 11 illustrates the accu-
racy of the proposed calcification segmentation. It can be seen P (t ≤ T < t + ∆t|T ≥ t)
hi (t) ≡ lim
that the results of the automated framework vary across the ∆t→0 ∆t
(35)
dataset. Table V shows the mean and the standard deviation p(t)
= , i = 1, 2 ,
of the Jaccard index denoted by J¯ and std(J), respectively. 1 − P (t)
 12

(a) (b) (c)

(d) (e) (f) (g) (h) (i)

Fig. 11. (a) The manual annotations from a trained radiologist. (b) The segmentation result of the automated framework shown as the conditional mean of
the final samples. (c) The overlay of Fig. 11(a) and Fig. 11(b). The calcifications are colored as follows: Yellow denotes true positives, red false negatives and
blue false positives. (d)-(e) A good calcification segmentation result where the manually annotated calcification is successfully detected. (f)-(g) A calcification
segmentation result with zero Jaccard index. (h)-(i) A poor calcification segmentation result with a lot of false negatives – independent of the vertebral shift.
Note that we have cropped images (d)-(i) for better visualization. The full images have the same rim artifacts as (a)-(c).

TABLE VI
H AZARD RATIOS FOR THE EPIPF BASELINE STUDY III. D ISCUSSION
1
R 5y We have presented a Bayesian framework for automatically
R(0) R(t)dt
5y 0 scoring the CVD mortality risk based on standard lumbar
Radiologist 6.8 4.5 radiographs. The framework has been validated by measuring
This study (Semibl ) 4.8 2.8
the framework’s ability to distinguish high-risk CVD patients
This study (Autobl ) 4.5 2.4
(AC24 score > 3.5) from others (AC24 score ≤ 3.5). On
the EPIPF baseline images, the framework’s mean hazard
ratio for the first five years was 2.4, which compares to the
and the hazard ratio
discriminative power of a manually computed AC24 score
h2 (t) (mean hazard ratio of 4.5). In other words, a patient with an
R(t) = . (36)
h1 (t) automated AC24 score > 3.5 has according to the mean hazard
We report the instantaneous hazard ratio at t = 0 and ratio over 5 years a 2.4 times higher risk of dying of CVD
the 5-year-mean-hazard-ratio between the two groups for the than a patient with an AC24 score ≤ 3.5. This result suggests
automated, semi-automated, and manual AC24 scoring (Table that the presented framework can be useful for efficient and
VI). Fig. 12 illustrates that the manual scores are better than objective risk assessment of CVD mortality.
the automated one, but also that all scorings manifest an The framework could for example be employed during
increased CVD death risk in the group of higher AC24 scores. osteoporosis (OP) screening, since a high risk for OP relates to

a high risk of CVD events [32]. It is known that both diseases
are highly prevalent in postmenopausal women, who do not
take estrogens, or have other risk factors for OP. Hence, the
same lateral lumbar radiograph could be used to measure the
risk for two highly prevalent public diseases.
The accuracy of the automated framework is good for most
of the cases but failure cases may occur due to, for instance,
level shifting in the vertebrae segmentation. The influence of
vertebral shifts was quantified by analyzing a semi-automated
variant of our framework. After manually fixing one vertebral
landmark, the mean hazard-ratio increased by 17% to 2.8.
Thus, future work will focus on reducing the number of
shifts by devising more discriminative vertebrae features and
explicitly modeling surrounding anatomical structures.
The vertebrae segmentation of the proposed framework
competes with the state-of-the art. The vertebrae SMC sampler
achieved accurate segmentations for most of the compared
radiographs – even on the baseline scans, which are degraded
by a lot of projection artifacts and image noise.
The proposed automated aorta segmentation is nearly as
accurate as the semi-automated approach in [5]. Most of the
segmentation errors can be explained by vertebral level shifting
and limited flexibility of the aorta shape model. Certain rare
cases are not well represented by the shape model. However,
we suspect that this problem could be alleviated by taking the
non-linearity of the shape manifold into account.
The mean Jaccard index between automatically and manu-
ally computed calcifications is not very high, but sufficient for
computing an AC24 score [21]. In particular on the baseline
dataset, where no or only few calcifications are visible, a
difference by one calcification leads to a small Jaccard index,
whereas the corresponding AC24 score is hardly affected.
It is challenging to devise a segmentation algorithm which
matches the accuracy of trained radiologists. One problem is
that standard radiographs contain an abundance of structures
with similar local appearance as calcifications. Contextual
knowledge is often indispensable for segmenting calcifications,
but most prior assumptions are not sufficient for describing the
full variation in the testing set. The calcification location prior,
for instance, raises the expected segmentation accuracy, but
fails in cases, where calcifications occur at unlikely locations.
To lower the dependence on a location and a spatial prior, we
currently work on methods that learn data-dependent features.
We expect that these features will characterize the calcification
appearance better than the generic multiscale local jet features.
Another limitation is that the manual annotations can only
be conceived as a guideline, rather than a ground truth. Ex-
periments showed that the Jaccard index between two trained
radiologists was only 0.51. To build a segmentation algorithm
that is potentially more precise than a trained radiologist, it
would be necessary to access a more reliable ’ground truth’,
such as annotations from registered X-ray/CT scans.
Although a trustworthy ground truth is required for im-
proving the calcification segmentations, the main performance
measure remains the prediction of CVD mortality. In this
paper, we have presented an automated framework that is able
to score standard radiographs, so as to discriminate high-risk
patients from others. Further developments of the segmentation
13
techniques will inevitably improve the accuracy and robustness
of the system, but already the current results demonstrate the
framework’s potential for better diagnosis and prognosis of
high-risk CVD patients.
ACKNOWLEDGMENT
The authors would like to thank the Center for Clinical
and Basic Research for providing scans and radiographic
readings. We gratefully acknowledge the funding from the
Danish Research Foundation (Den Danske Forskningsfond)
supporting this work. We also thank the anonymous reviewers
for their useful comments.
R EFERENCES
[1] S. Allender, P. Scarborough, V. Peto, and M. Rayner. European
cardiovascular disease statistics 2008.
[2] Y. Z. Bagger, L. B. Tankó, P. Alexanderson, H. B. Hansen, G. Qin, and
C. Christiansen. The long-term predictive value of bone mineral density
measurements for fracture risk is independent of the site of measurement
and the age at diagnosis: results from the prospective epidemiological
risk factors study. In Osteoporosis International, volume 17, pages 471–
477. Springer, 2006.
[3] G. J. F. Botev. Zdravko I. and K. D. P. Kernel density estimation via
diffusion. Annals of Statistics, 38(5):2916–2957, 2010.
[4] L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
[5] M. D. Bruijne. Shape particle guided tissue classification. In P. Golland
and D. Rueckert, editors, In Proc. Mathematical Methods in Biomedical
Image Analysis (MMBIA 2006), 2006.
[6] M. d. Bruijne and M. Nielsen. Image segmentation by shape particle
filtering. In Proc. International Conference on Pattern Recongition (ICPR
2004), volume 3, pages 722–725, Washington, DC, USA, 2004. IEEE
Computer Society.
[7] W. L. Buntine. Operations for learning with graphical models. Journal
of Artificial Intelligence Research , 2:159–225, 1994.
[8] J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational
and Psychological Measurement, 20(1):37–46, April 1960.
[9] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Training
models of shape from sets of examples. In Proc. British Machine Vision
Conference (BMVC 1992), pages 9–18. Springer-Verlag, 1992.
[10] M. de Bruijne and M. Nielsen. Shape particle filtering for image
segmentation. In C. Barillot, D. R. Haynor, and P. Hellier, editors,
Proc. Medical Image Computing and Computer Assisted Intervention
(MICCAI 2004), volume 3216 of Lecture Notes in Computer Science,
pages 168–175. Springer, 2004.
[11] P. Del Moral, A. Doucet, and A. Jasra. Sequential monte carlo
samplers. Journal of the Royal Statistical Society: Series B (Statistical
Methodology), 68(3):411–436, June 2006.
[12] A. Doucet, N. de Freitas, and N. Gordon. Sequential Monte Carlo
methods in practice. Springer-Verlag New York, Inc., 2001.
[13] L. Fischer, R. Donner, F. Kainberger, and G. Langs. Automatic
region template generation for shape particle filtering based image
segmentation. In Proc. Probabilistic Models for Medical Image Analysis
(PMMIA 2009), in conjunction with MICCAI 2009 , pages 289–300,
2009.
[14] L. Florack, B. Ter Haar Romeny, M. Viergever, and J. Koenderink. The
gaussian scale-space paradigm and the multiscale local jet. International
Journal of Computer Vision, 18:61–75, April 1996.
[15] M. Ganz, M. Nielsen, and S. S. Brandt. Patch-based generative shape
model and mdl model selection for statistical analysis of archipelagos.
In Proc. Machine Learning in Medical Imaging (MLMI), in conjunction
with MICCAI 2010, pages 34–41, 2010.
[16] H. K. Genant, C. Y. Wu, C. van Kuijk, and M. C. Nevitt. Vertebral
fracture assessment using a semiquantitative technique. Journal of bone
and mineral research, 8(9):1137–1148, 1993.
[17] A. J. Gray, J. W. Kay, and D. M. Titterington. An empirical study of
the simulation of various models used for images. IEEE Trans. Pattern
Analalysis and Machine Intelligence , 16:507–513, May 1994.
[18] M. Hansson, S. Brandt, and P. Gudmundsson. Bayesian probability maps
for evaluation of cardiac ultrasound data. In Probabilistic Models for
Medical Image Analysis (PMMIA 2009), in conjunction with MICCAI
2009, 2009.
 14

[19] J. E. Iglesias and M. de Bruijne. Semi-automatic segmentation of

vertebrae in lateral x-rays using a conditional shape model. Academic
Radiology, 14:1156–1165, 2007.
[20] P. Jaccard. Étude comparative de la distribution florale dans une portion
des Alpes et des Jura. Bulletin del la Société Vaudoise des Sciences
Naturelles, 37:547–579, 1901.
[21] L. I. Kauppila, J. F. Polak, L. A. Cupples, M. T. Hannan, D. P. Kiel, and
P. W. Wilson. New indices to classify location, severity and progression
of calcified lesions in the abdominal aorta: a 25-year follow-up study.
Astherosclerosis, (132):245 –250, 1997.
[22] R.-E. W. Kavey, S. R. Daniels, R. M. Lauer, D. L. Atkins, L. L.
Hayman, and K. Taubert. American Heart Association Guidelines for
Primary Prevention of Atherosclerotic Cardiovascular Disease Beginning
in Childhood. Circulation, 107(11):1562–1566, March 2003.
[23] N. I. Parikh, S.-J. Hwang, M. G. Larson, L. A. Cupples, C. S. Fox,
E. S. Manders, J. M. Murabito, J. M. Massaro, U. Hoffmann, and C. J.
O’Donnell. Parental occurrence of premature cardiovascular disease
predicts increased coronary artery and abdominal aortic calcification
in the framingham offspring and third generation cohorts. Circulation,
116(13):1473–1481, 2007.
[24] K. Petersen, M. Nielsen, and S. S. Brandt. Conditional point distribution
models. In B. Menze, G. L. G, Z. Tu, and A. Criminisi, editors,
Proc. Medical Computer Vision 2010, in conjuction with MICCAI 2010,
Beijing, China., Heidelberg, September 2010. Springer-Verlag.
[25] K. Petersen, M. Nielsen, and S. S. Brandt. A static smc sampler
on shapes for the automated segmentation of aortic calcifications. In
K. Daniilidis, P. Maragos, and N. Paragios, editors, Proc. European
Conference on Computer Vision (ECCV 2010) , London, UK, 2010.
Springer-Verlag.
[26] P. D. Reaven and J. Sacks. Coronary artery and abdominal aortic cal-
cification are associated with cardiovascular disease in type 2 diabetes.
In Diabetologia, volume 48, pages 379–385. Springer-Verlag, 2005.
[27] C. P. Robert. The Bayesian Choice: From Decision-Theoretic
Foundations to Computational Implementation (Springer Texts in
Statistics). Springer Verlag, New York, 2nd edition, June 2007.
[28] M. G. Roberts, T. F. Cootes, and J. E. Adams. Automatic segmentation
of lumbar vertebrae on digitised radiographs using linked active ap-
pearance models. In Proc. Medical Image Understanding and Analysis
(MIUA 2006), volume 6, pages 120–124, 2006.
[29] M. G. Roberts, T. F. Cootes, and J. E. Adams. Vertebral morphometry:
semiautomatic determination of detailed shape from dual-energy x-ray
absorptiometry images using active appearance models. Investigative
radiology, 41(12):849–859, 2006.
[30] M. G. Roberts, T. F. Cootes, E. Pacheco, T. Oh, and J. E. Adams.
Segmentation of lumbar vertebrae using part-based graphs and active
appearance models. In Proc. Medical Image Computing and Computer
Assisted Intervention (MICCAI 2009), pages 1017–1024, Berlin, Hei-
delberg, 2009. Springer-Verlag.
[31] V. L. Roger, A. S. Go, D. M. Lloyd-Jones, R. J. Adams, J. D. Berry,
T. M. Brown, M. R. Carnethon, S. Dai, G. de Simone, E. S. Ford, C. S.
Fox, H. J. Fullerton, C. Gillespie, K. J. Greenlund, S. M. Hailpern,
J. A. Heit, P. M. Ho, V. J. Howard, B. M. Kissela, S. J. Kittner, D. T.
Lackland, J. H. Lichtman, L. D. Lisabeth, D. M. Makuc, G. M. Marcus,
A. Marelli, D. B. Matchar, M. M. McDermott, J. B. Meigs, C. S. Moy,
D. Mozaffarian, M. E. Mussolino, G. Nichol, N. P. Paynter, W. D.
Rosamond, P. D. Sorlie, R. S. Stafford, T. N. Turan, M. B. Turner, N. D.
Wong, J. Wylie-Rosett, on behalf of the American Heart Association
Statistics Committee, and S. S. Subcommittee. Heart disease and stroke
statistics–2011 update: A report from the american heart association.
Circulation, 2010.
[32] D. E. Warburton, C. W. Nicol, S. N. Gatto, and S. S. Bredin. Cardiovas-
cular disease and osteoporosis: Balancing risk management. In Vascular
Helath and Risk Management, volume 3, pages 673–689. Dovepress,
October 2007.
[33] P. W. F. Wilson, L. I. Kauppila, C. J. O’Donnell, D. P. Kiel, M. Hannan,
J. M. Polak, and A. Cupples. Abdominal aortic calcific deposits are an
important predictor of vascular morbidity and mortality. Circulation,
(103):1529–1534, 2001.
[34] J. C. M. Witteman, F. J. Kok, J. L. C. M. van Saase, and H. A. Valken-
burg. Aortic calcification as a predictor of cardiovascular mortality.
Lancet, 328(2):1120–1122, 1986.