Medical Image Analysis 16 (2012) 38–49

Contents lists available at ScienceDirect

Medical Image Analysis

journal homepage: www.elsevier.com/locate/media

Registration of 3D trans-esophageal echocardiography to X-ray fluoroscopy

using image-based probe tracking
Gang Gao a,⇑, Graeme Penney a, Yingliang Ma a, Nicolas Gogin b, Pascal Cathier b, Aruna Arujuna a,
Geraint Morton a, Dennis Caulfield d, Jaswinder Gill d, C. Aldo Rinaldi d, Jane Hancock d, Simon Redwood d,
Martyn Thomas d, Reza Razavi a, Geert Gijsbers c, Kawal Rhode a
a
Divsion of Imaging Sciences & Biomedical Engineering, King’s College London, UK
b
Medisys Research Group, Philips Healthcare, Paris, France
c
Business Unit iXR/Business Program EP, Philips Healthcare, Best, The Netherlands
d
Department of Cardiology, Guy’s & St. Thomas’ Hospitals NHS Foundation Trust, London, UK

a r t i c l e i n f o a b s t r a c t

Article history: Two-dimensional (2D) X-ray imaging is the dominant imaging modality for cardiac interventions. How-
Received 22 June 2010 ever, the use of X-ray fluoroscopy alone is inadequate for the guidance of procedures that require soft-
Received in revised form 20 April 2011 tissue information, for example, the treatment of structural heart disease. The recent availability of
Accepted 4 May 2011
three-dimensional (3D) trans-esophageal echocardiography (TEE) provides cardiologists with real-time
Available online 12 May 2011
3D imaging of cardiac anatomy. Increasingly X-ray imaging is now supported by using intra-procedure
3D TEE imaging. We hypothesize that the real-time co-registration and visualization of 3D TEE and X-
Keywords:
ray fluoroscopy data will provide a powerful guidance tool for cardiologists. In this paper, we propose
Image registration
Cardiac intervention
a novel, robust and efficient method for performing this registration. The major advantage of our method
Trans-esophageal echocardiography is that it does not rely on any additional tracking hardware and therefore can be deployed straightfor-
X-ray fluoroscopy wardly into any interventional laboratory. Our method consists of an image-based TEE probe localization
algorithm and a calibration procedure. While the calibration needs to be done only once, the GPU-accel-
erated registration takes approximately from 2 to 15 s to complete depending on the number of X-ray
images used in the registration and the image resolution. The accuracy of our method was assessed using
a realistic heart phantom. The target registration error (TRE) for the heart phantom was less than 2 mm.
In addition, we assess the accuracy and the clinical feasibility of our method using five patient datasets,
two of which were acquired from cardiac electrophysiology procedures and three from trans-catheter
aortic valve implantation procedures. The registration results showed our technique had mean registra-
tion errors of 1.5–4.2 mm and 95% capture range of 8.7–11.4 mm in terms of TRE.
Ó 2011 Elsevier B.V. All rights reserved.

1. Introduction be performed at high frame rates (up to 30 frames per second)

Minimally-invasive cardiovascular interventions are carried out
for the diagnosis and treatment of a broad range of cardiovascular
diseases. These types of procedures are increasingly popular when
compared to their more invasive counterparts because there is less
morbidity to the patient with similar clinical outcomes of success.
Examples of these procedures include those carried out for the re-
pair of structural heart disease and cardiac electrophysiology (EP)
procedures. The interventional devices, for example, catheters,
are designed to be X-ray visible and can be seen throughout the
part of their length that lies in the X-ray field of view (FOV).
Two-dimensional (2D) X-ray imaging is the dominant imaging
modality for guiding cardiac interventions. Typically, imaging can
⇑ Corresponding author. Tel.: +44 02071888376.
E-mail address: gaogang@gmail.com (G. Gao).
and therefore the cardiac motion and the motion of interventional
devices do not cause significant motion artifacts in the acquired
images. However, the use of X-ray fluoroscopy alone is inadequate
for the guidance of procedures that require soft-tissue information,
for example, the treatment of structural heart disease (Silverstry
et al., 2009). In addition, exposing patients, especially pediatric pa-
tients with congenital defects, to ionizing radiation carries a signif-
icant risk (Kovoor et al., 1998; Modan et al., 2000). On-going
research is actively seeking to reduce the use of, or even replace,
X-ray fluoroscopy in cardiac interventional procedures, especially
for pediatrics (Razavi et al., 2003). Pre-operatively acquired mag-
netic resonance imaging (MRI) and computed tomography (CT)
have been used with X-ray fluoroscopy to improve the guidance
of cardiac catheterization procedures (Rhode et al., 2003, 2005;
Yu et al., 2005; De Buck et al., 2005; Sra et al., 2007). The image
registration of pre-procedure MRI/CT and intra-procedure X-ray

1361-8415/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved.
doi:10.1016/j.media.2011.05.003
 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49
can be achieved either by using prior calibration (Rhode et al.,
2003, 2005; Yu et al., 2005), manual registration (De Buck et al.,
2005), or automatic approaches (Sra et al., 2007). Recently, the
use of three-dimensional (3D) rotational X-ray angiography (3D
RXA) (Knecht et al., 2010; Manzke et al., 2010) has been shown
to be useful for cardiac interventional guidance. This imaging
modality is perfectly suited to efficient clinical work flow since
the 3D image data are acquired just prior to the intervention with
the patient lying on the same X-ray table. Therefore, the registra-
tion of the 3D data to the 2D fluoroscopy data is implicit. However,
with the limitation of the current technology the image quality and
FOV of 3D RXA are considerably lower than those of MRI and CT.
The use of pre-procedural MR/CT and 3D RXA imaging will pro-
duce roadmap images that are static and do not update with the
intra-procedural situation. Additional steps are required to com-
pensate for intra-procedural deformations, for example, those
caused by respiratory motion of the heart (King et al., 2009). In
contrast to MR and CT imaging, 3D echocardiography (echo) is a
real-time imaging modality that can be readily used in the catheter
laboratory environment. Because of its relatively low image quality
and small FOV, 3D echo has been used as an intermediate imaging
modality to register X-ray fluoroscopy with high resolution imag-
ing modalities such as MRI and CT (King et al., 2010). Although pre-
vious works have successfully registered 3D echo with other
imaging modalities such as MRI (King et al., 2010; Roche et al.,
2001; Penney et al., 2004; Wein et al., 2009), the registration be-
tween echo and X-ray fluoroscopy still relied on prior calibration
and tracking. With recently emerging technologies such as 3D
trans-esophageal echo (TEE), the image quality and resolution of
echo has improved considerably. In comparison to trans-thoracic
echo, TEE has unique access to structures such as the aorta, the
valves of the heart and both atria. This makes TEE an ideal solution
for the guidance of cardiac interventions. Recently, the clinical fea-
sibility of 3D TEE was evaluated for guiding a variety of cardiac
interventional procedures including atrial septal defect closure,
patent foramen ovale closure, mitral valve/aortic valve repair and
EP procedures (Barker et al., 2008; Mackensen et al., 2008). TEE ap-
pears to be a promising imaging modality for cardiac interventions
and its routine clinical use is rapidly increasing. It is already well-
established in the clinical work flow of several interventions, such
as minimally-invasive valve implantation procedures.
Combining the soft-tissue information from TEE with the excel-
lent device visualization of X-ray fluoroscopy could prove to be a
powerful combination in the catheter laboratory. The purpose of
this study was to develop a practical technique to combine 3D
TEE and X-ray fluoroscopy images for the guidance of cardiac cath-
eterization procedures. A fast and robust image registration of 3D
TEE and X-ray fluoroscopy was developed for this purpose. Previous
approaches relied on the use of an electromagnetic (EM) tracking
device (Gao et al., 2009; Jain et al., 2009). However, the use of the
EM tracker has several disadvantages including the requirement
of additional hardware, the requirement of modifications to the
ultrasound probe, and the sensitivity of these systems to metallic
interference. Neither of the studies reported in (Jain et al., 2009;
Gao et al., 2009) involved clinical evaluations but were limited to
phantoms, therefore, the performance of EM tracking in the routine
clinical work flow is unclear. Our method was based upon an im-
age-based 2D–3D registration algorithm. It does not rely on track-
ing hardware or modification to the TEE probe and therefore can
be deployed straightforwardly to any cardiac catheterization labo-
ratory. Preliminary results were reported in (Gao et al., 2010). In
this paper, we will describe a more thorough study to examine
the performance of the 2D–3D registration algorithm and the over-
all TEE and X-ray image registration using both phantom and clin-
ical data. Clinical data (n = 5), which were acquired from two
different types of cardiac intervention procedures, including car-
39
diac EP procedures and trans-catheter aortic valve implantations
(TAVI), were used to examine the clinical feasibility of the proposed
algorithm. Although all the clinical data were currently processed
off-line, a clinical workflow was proposed to demonstrate how
our technique will be used live in the clinical environment.
2. Methods
2.1. Localization of the TEE probe
Our method is based on the localization of the TEE probe. Since
the addition of tracking devices, such as an EM tracker, is not ideal
for the clinical cardiac catheterization environment, the key chal-
lenge in our method is to design an efficient and robust image-
based TEE probe tracking algorithm. Fig. 1a and b shows the 3D
TEE probe used in this study and an X-ray image acquired during
an aortic valve implant procedure, clearly showing the TEE probe.
Our hypothesis is that by using one or more X-ray images, the po-
sition and the orientation of the TEE probe can be automatically
determined from the X-ray data.
2.1.1. 3D reconstruction of the TEE probe
Although the TEE probe is designed to be flexible, its transducer
is encapsulated in a rigid head. For simplicity, the rigid head of the
TEE probe is referred to as ‘‘the TEE probe’’ in the following text.
The first step of our localization algorithm is to reconstruct a pre-
cise 3D model of the TEE probe. Our approach is to use a nano-CT
system (nano-Pet™/CT, Mediso Ltd., Budapest, Hungary) which is
capable of reconstructing ultra-high-resolution 3D volumes.
Fig. 1c shows the nano-CT reconstructed TEE probe (matrix:
342  342, resolution = 0.2  0.2  0.2 mm3). The 3D reconstruc-
tion of the TEE probe does not need to be repeated unless a differ-
ent model of TEE probe is used.
2.1.2. 2D–3D image registration
In order to determine the position of the TEE probe in C-arm
space (i.e. the local 3D coordinate system of the X-ray system),
the nano-CT volume of the TEE probe was registered to the X-ray
images using a 2D–3D image registration algorithm. Intensity-
based 2D–3D image registration has been extensively studied
(Penney et al., 1998; Hipwell et al., 2003; van de Kraats et al.,
2005; Turgeon et al., 2005). It has been used to register pre-opera-
tive CT or MR volume of bones, blood vessels and medical devices
to intra-operative X-ray images (Hipwell et al., 2003; Turgeon et
al., 2005). Fig. 2 shows a typical workflow of an intensity-based
2D–3D registration algorithm. During the registration process,
the algorithm repeatedly repositions the 3D volume in space and
compares its projection, called the digitally reconstructed radiograph
(DRR), with one or multiple X-ray image(s). At each iteration, an
image comparison metric is used to calculate the similarity be-
tween the X-ray image(s) and the DRR. The three translation and
three rotation parameters are changed according to the similarity
measurement. The registration process continues until the similar-
ity between the X-ray image(s) and the DRR is maximized.
In this study, we implemented an intensity-based 2D–3D regis-
tration algorithm similar to the one described in (Penney et al.,
1998). This work shows gradient difference (GD) and pattern
intensity (PI) are more robust and accurate than other intensity-
based similarity measurements, such as mutual information or
cross correlation. Both GD and PI are largely unaffected by the
presence of soft tissue and thin line structures, such as guide wires,
stents and other interventional devices that appeared in the X-ray
fluoroscopy. In this study, GD was used as the similarity measure-
ment function in our 2D–3D registration algorithm. A classic Pow-
ell function was used as the optimizer. The speed of the original
40 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

Fig. 1. (a) The 3D TEE probe used (X7-2t, Philips Healthcare, Andover, Boston, USA); (b) A typical X-ray fluoroscopy image of the TEE probe; (c) A high-resolution 3D nano-CT
scan reveals the contours and the internal structure of the TEE transducer. The 3D nano-CT volume was used to localize the 3D position of the TEE probe in X-ray space.

algorithm described in (Penney et al., 1998) was relatively slow be-

cause the generation of DRRs is computationally expensive. It
could take minutes or longer to register the nano-CT volume to a
standard X-ray image. Our implementation overcomes this issue
by taking advantage of the latest GPU technology. The DRR recon-
struction time was reduced to less than 10 ms for each estimated
pose. By using a standard computer workstation equipped with
an Intel Quad-Core CPU (2.66 GHz), 4 GB RAM and an NVIDIA Ge-
Force GTX 280 graphics card, the overall registration time between
the nano-CT volume and two standard X-ray images is from 1 to
15 s, depending on the initial probe position and the size and num-
ber of X-ray images used. The evaluation of registration speed will
be presented in Sections 3.2.1 and 4.1.2.
It is well acknowledged in previous studies that 2D–3D registra-
tion using a single X-ray image can potentially introduce consider-
able registration error in the direction of the X-ray beam. To
minimize such error, two X-ray images acquired from different an-
gles must be used. Our 2D–3D registration algorithm can process
one or multiple X-ray images simultaneously depending on the
requirement of the study. In-depth discussion related to this topic
will be presented in Sections 4.1 and 5.
The authors acknowledge that the registration of TEE and X-ray
is a form a 2D–3D registration. However for simplicity and clarity,
the ‘‘2D–3D registration’’ mentioned in the following text refers
specifically to the intensity-based 2D–3D image registration of
nano-CT and X-ray.
Fig. 2. The workflow of an intensity-based 2D–3D registration algorithm.

2.2. Image registration of 3D TEE and X-ray fluoroscopy
The transformation matrix, TUS_img?X-ray_img, which transforms
from 3D TEE image space to 2D X-ray image space consists of a ri-
gid body transformation matrix, Trigid, and a projection matrix, Tproj,
T US img!Xray img ¼ T proj T rigid :
arm space. This matrix is generated by the 2D–3D registration algo-
rithm that positions the nano-CT volume in C-arm space. TUS_img?
tracked relates the position of the 3D TEE images with the position
of the TEE probe in nano-CT space. This is the TEE probe calibration
matrix and can be calculated pre-procedurally using a specifically
ð1Þ designed calibration phantom.

The projection matrix, Tproj, transforms from 3D C-arm space, i.e.
the local coordinate system of the X-ray system, to 2D X-ray image
space. This can be calculated by using the intrinsic parameters of
the X-ray system that are determined from a system calibration
and are available from the DICOM image header (Hawkes et al.,
1987). Trigid can be decomposed into two matrices
T US ¼ T proj T tracked!C-arm T US
img!Xray img img!tracked
where in our case, Ttracked?C-arm is the transformation matrix from
the 3D tracked coordinate system, i.e. the nano-CT space, to 3D C-
2.3. TEE probe calibration
The TEE probe calibration procedure aims to determine
TUS_img?tracked (Mercier et al., 2005). The calibration phantom
consists of a 9-L water tank and two thin metal wires. Nine metal
landmarks which were visible in both X-ray and ultrasound were
ð2Þ placed on the wires. The TEE probe was rigidly fixed beneath the
wires during data acquisition. X-ray images were acquired from left
anterior oblique (LAO) 45°, right anterior oblique (RAO) 45° and
posterio-anterior (PA) projections using a Philips Allura Xper FD10
 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 41

C-arm X-ray system, which has an internal mechanism to track the

C-arm position in real-time. Simultaneously an echo volume was ac-
quired in full volume mode, giving the maximal volume coverage
possible with the TEE probe. The following steps were adopted to
determine TUS_img?tracked: (a) a quick manual registration was per-
formed to align the nano-CT volume with the X-ray images acquired
from PA & LAO; and (b) the automatic 2D–3D registration algorithm
described in Section 2.2 was then utilized to refine the alignment
simultaneously in both views. This registration ensured an accurate
localization of the TEE probe in 3D C-arm space, i.e. determination of
Ttracked?C-arm. The third X-ray image, which was acquired from RAO
45° was used to confirm the accuracy of the TEE probe localization.
The 3D positions of the nine landmarks, PphantomUS, were identi-
fied manually from the TEE image data. The landmarks were also
clearly visible in the X-ray images. By manually defining the 2D posi-
tion of the landmarks in the PA and LAO 45° X-ray images, their 3D
positions in C-arm space, PphantomC-arm, could be reconstructed using
back-projection (Hawkes et al., 1987). The calibration procedure was
repeated for three different probe positions. A classic hill-climbing
optimization algorithm was employed to find TUS_img?tracked by
minimizing the Euclidean distance error, e, given by
 
e ¼ T US img!tracked PphantomUS  T 1 
tracked!C-arm PphantomC-arm  ð3Þ

In order to validate the accuracy of the calibration, a further 2

TEE volumes were acquired of the calibration phantom along with
X-ray images in the PA and LAO 45° views. For these data, we used
Eq. (3) to compute the mean 3D Euclidean distance errors in
tracked nano-CT space.

Fig. 3. The proposed workflow of the X-ray and TEE image registration in clinical
2.4. Clinical workflow procedures.

The objective of this study was to develop a practical solution to
integrate X-ray and 3D echo for the guidance of cardiac interven-
tional procedures. We propose a clinical workflow to demonstrate
how the proposed TEE and X-ray registration algorithm can be
used in clinical environments. The workflow consists of seven com-
ponents (Fig. 3), some of which are currently manual but could be
automated in the future, for example, the initial 2D–3D registra-
tion and the failure detection for the probe localization. The probe
tracking component, which is designed to compensate the jittering
motion of the TEE probe caused by cardiac and respiratory motion,
is not available at present. It is an open research topic. One solution
would be to speed up the intensity-based 2D–3D registration by
incorporating more sophisticated implementations such as multi-
ple-resolution techniques and advanced GPU computing. More de-
tails about the jittering motion of the TEE probe will be discussed
in Section 3.2.3. Technically, the 2D–3D registration algorithm can
work either with mono-plane or bi-plane X-ray. The workflow does
not restrict the number of X-ray images used in the 2D–3D regis-
tration. The decision on whether mono-plane or bi-plane X-ray
images will be used should be made based on the clinical require-
ments and how the registration results will be utilized.
anthropomorphic external and internal anatomy including left/
right ventricles, left/right atria and the valves (Fig. 4).
For TEE acquisition, we used an iE33 3D real-time echo system
with an X7-2t 3D TEE probe (Philips Healthcare, Andover, Boston,
USA). The TEE probe was placed on the acoustic surface of the heart
phantom. Six TEE volumes were acquired to cover different
sections of the phantom by varying the TEE probe position. Nine
X-ray and MRI-visible markers (Multi-modality Radiographic
Marker, IZI Medical Products, Baltimore, USA) were placed on the
surface of the phantom. The MR volume of the heart phantom
was acquired a day before the TEE and X-ray scans in the catheter
laboratory using a Philips Achieva 1.5T MR scanner. The MR proto-
col consisted of a 3D BTFE scan with the following parameters:
512  512 matrix, 180 slices, resolution = 0.96  0.96  1.0 mm3,
TR = 8.42 ms, TE = 4.2 ms, flip angle = 90°. Between the MRI scan
and the TEE scan, the heart phantom was carefully stored in order
to avoid displacement of the markers. For X-ray image acquisition,

3. Experiments

3.1. Imaging equipment and data acquisition

Experimental data were acquired from the TEE probe calibra-

tion phantom, a realistic heart phantom and five patients undergo-
ing cardiac interventional procedures.

3.1.1. Realistic heart phantom

We evaluated the accuracy of our method using a realistic heart
phantom (Ultrasound Heart Phantom, Computerized Imaging Ref- Fig. 4. The heart phantom experiment was performed in the catheter laboratory.
erence Systems, Inc., Virginia, USA). The phantom has completely The picture shows the system setup of the experiment.
42 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49
we used a Philips Allura Xper FD10 C-arm X-ray system, the same
X-ray system used for the TEE probe calibration.
3.1.2. Cardiac EP procedures
We collected data from two cardiac EP procedures. Both of the
patients had left atrial flutter and were under general anesthesia
(GA) during the procedures. For the first patient, TEE and X-ray
data were acquired after two deca-polar catheters were inserted
into the right atrium (RA), one forming a loop along the endocar-
dial surface of RA and the other inserted into the coronary sinus
(CS). Both catheters were visible in the TEE volume. For the second
patient, a trans-septal puncture was performed to gain access to
the left atrium (LA). TEE volumes were acquired after a lasso cath-
eter and an ablation catheter were inserted into LA. The movement
of the C-arm was limited by other equipment such as the life sup-
port system and the ultrasound scanner. X-ray images covering 4–
5 cardiac cycles were acquired from PA and either RAO 30° or LAO
30° projections. Cardiac and respiratory phases were estimated
using the motion of the interventional devices such as the cathe-
ters (Brost et al., 2010; Klemm et al., 2007). X-ray images acquired
during the same cardiac and respiratory phases were used in the
2D–3D registration.
3.1.3. Trans-catheter aortic valve implant procedures
TAVI data were collected from three patients with degenerative
aortic stenosis. All the three procedures were performed in a GE
catheter laboratory equipped with a GE Innova 2100IQ C-arm X-
ray system. Similar to the Philips Allura Xper FD10, this GE X-ray
system can precisely track its C-arm position automatically. The
TEE data were acquired using the same echo system and TEE probe
described above. In the first two TAVI cases, the replacement
valves were delivered using trans-femoral approach. In the third
TAVI case, the replacement valve was delivered using trans-apical
approach. X-ray images and TEE volumes were acquired in the
same way as the data acquisition in the EP procedures after the
replacement valve reached the deployment site.
3.2. Evaluation of the nano-CT and X-ray image registration
By using our method, a successful TEE and X-ray image registra-
tion depends on a fast and robust 2D–3D nano-CT and X-ray image
registration algorithm. In this experiment, we evaluate the robust-
ness and the speed of the 2D–3D registration algorithm using stan-
dard methodologies presented in previously published literature
(van de Kraats et al. 2005).
3.2.1. Experiment 1 – Capture range and speed of the 2D–3D
registration
The capture range was defined as the largest starting misalign-
ment at and below which registration was successful 95% of the time
(Huang et al., 2009; Shekhar and Zagrodsky, 2002). To evaluate the
capture range, a reference transformation matrix was introduced
for each of the testing datasets. The reference transformation matrix
was prepared by the following procedures:
(a) the automatic 2D–3D registration, as described in Section
2.1, was employed to generate a starting position to align
the nano-CT volume with bi-plane X-ray images;
(b) the registration was then refined using manual expert inter-
action allowing all six degrees of freedom during careful
visual examination of the result to ensure the closest possi-
ble match;
(c) steps (a) and (b) were repeated three times resulting in three
transformation matrices. The reference matrix was con-
structed using the mean parameters of these three matrices.
The starting positions were created as follows:
(a) For each of the six degrees of freedom, a range was deter-
mined. Within the ranges, transformation matrices were
generated using all possible combinations of the six DOFs.
The center of the rotation is the center of the nano-CT vol-
ume. New positions of the nano-CT volume were generated
by transforming the original nano-CT volume using the
matrices;
(b) For each of the newly generated nano-CT positions, its cor-
ner point distance to the position of the original nano-CT
volume was calculated as the mean target registration error
(Eq. (4)). The new nano-CT positions were sorted using their
mTRE to the original nano-CT position;
(c) Intervals were chosen for the starting position distance, for
example, 0–1 mm, 1–2 mm. For each interval, 100 new
nano-CT positions were selected randomly (uniformly dis-
tributed) as the starting positions.
For the calculation of mTRE, the landmarks were the eight cor-
ners of the bounding box of the nano-CT volume. The testing data
used in this experiment were X-ray images acquired from the clin-
ical studies. For each of the seven testing datasets, 1400 starting
positions (1 mm 6 mTRE 6 15 mm) were examined to calculate
the capture range. The capture ranges using bi-plane X-ray images
and mono-plane X-ray images were studied separately
n  
1X  i i 
mTRE ¼ T registration Plandmark  T ref Plandmark  ð4Þ
n i¼1
where P denotes the landmarks defined on the eight corners of the
nano-CT volume. To evaluate the registration accuracy using mono-
plane X-ray images, the mean projection distance, mPD was calcu-
lated (van de Kraats et al., 2005)
n  
1X  i i 
mPD ¼  Mregistration Plandmark  M ref Plandmark   Sdet  F ð5Þ
n i¼1
where Mregistration and Mref are the final perspective projection
matrices for the registration and the reference. Sdet is the detector
size of the X-ray system. F is the magnification factor, which can
be calculated using Eq. (6)
Dsource!object
F¼ ð6Þ
Dsource!image
Dsource?object and Dsource?image are source to object distance and
source to image distance. Sdet, Dsource?object and Dsource?image can be
retrieved from either the live data stream from the X-ray
system or the header of the DICOM files. It should be noted that
Dsource?object is an approximated value and therefore the mean
projection distance calculated using Eq. (5) is also an approximated
value.mTRE has been previously proposed and accepted as a metric
to evaluate the capture range of 2D–3D registration in different
studies (Huang et al., 2009; van de Kraats et al., 2005). However,
it cannot clearly differentiate the errors in translation from the
errors in rotation. An additional step was used to examine the reg-
istration error in translations (mm) and rotations (°) separately by
using the magnitude of the rotations and the translations
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
t¼ t 2x þ t 2y þ t2z and h ¼ h2x þ h2y þ h2z ð7Þ
where t and h are the magnitude of the translations and the rota-
tions, tx, ty, tz and hx, hy, hz are the translations and rotations in x,
y and z directions.
The computation time of every registration was recorded to
analyze the efficiency of our 2D–3D registration algorithm.
 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 43

3.2.2. Experiment 2 – Evaluation of TEE and X-ray registration error

with regard to distance from the transducer
The 2D–3D registration error will be propagated into the overall
TEE and X-ray registration and small rotational errors can poten-
tially create a considerable error, especially for a target object at
the far side of the ultrasound cone. In this experiment, we exam-
ined how the error produced by the 2D–3D registration algorithm
affects the overall TEE and X-ray registration accuracy.
In the previous experiment, a considerable number of registra-
tions were performed. The registration can be considered as suc-
cessful if the end mTRE is not greater than a pre-defined
threshold. The matrices from all the successful registrations were
used to transform a number of landmarks defined within the TEE
FOV at different depths (Fig. 5). mTRE was calculated using Eq.
(4) to examine the potential target error at different depths in com-
parison to the reference matrix.
Fig. 6. MRI was used as a gold standard to assess the accuracy of the TEE and X-ray
3.2.3. Experiment 3 – Quantitative study of the error caused by image registration.
jittering motion of the TEE probe
As mentioned previously, the jittering motion of the TEE probe
3.3. Accuracy assessment of TEE and X-ray image registration
is unavoidable due to cardiac and respiratory motion. A real-time
probe tracking algorithm will minimize such error. We do not in-
3.3.1. Phantom study
tend to propose such a tracking algorithm in this paper. Instead,
Since the heart phantom is barely visible in X-ray, the MR
an experiment was set up to investigate the potential error caused
images of the heart phantom were used as a reference to assess
by the jittering motion of the TEE probe. The experiment consists
the accuracy of the TEE and X-ray image registration. The workflow
of three steps:
of the accuracy assessment is illustrated in Fig. 6. Firstly, the 3D
positions of the landmarks were determined manually in the MR
(1) acquire X-ray images of the TEE probe during clinical proce-
image. Secondly, positions of the corresponding landmarks were
dures to cover at least one respiratory cycle so that the cap-
determined manually from bi-plane X-ray images acquired from
tured jittering motion consists of both respiratory motion
LAO 45° and RAO 45° so that their 3D positions could be accurately
and cardiac motion;
reconstructed. A third X-ray image was acquired from PA view. The
(2) register the nano-CT volume to each of the X-ray images;
landmarks were projected back to all three X-ray images (LAO 45°,
(3) set the first frame of the X-ray sequence as the reference and
RAO 45° and PA) so that the 3D reconstruction accuracy can be
calculate the TEE probe motion from the second frame
estimated by calculating the distance between the projected land-
onwards in terms of mPD by:
marks and their manually defined positions. The MRI and C-arm
coordinate system registration was achieved using a landmark reg-
1X n  
ðM i Plandmark;j  M 1 Plandmark;j Þ  Sdet  F
mPDiðiP2Þ ¼ ð8Þ istration. To quantify the TEE and X-ray registration error, endocar-
n j¼1
dial surfaces of the heart phantom including left ventricle (LV),
right ventricle (RV), left atrium (LA), right atrium (RA) and left ven-
tricular papillary muscle (LVPM) were extracted and reconstructed
from the MR volume semi-automatically using an open source
software application ITK-Snap (Yushkevich et al. 2006). A total of
16–66 points were defined manually along the endocardial border
from the TEE volumes. Both the MR surfaces and the TEE points
were transformed to C-arm space. The registration error was as-
sessed both visually and by calculating the mean distance between
the TEE points and the MR surface. Point-to-surface distances do
not capture errors that are tangential to the surface but in our case
the surfaces were largely blob-like and therefore this should not af-
fect the results.
It should be noted that the point-to-surface distance represents
a combination of the MR to C-arm landmark registration error and
TEE to C-arm registration error.

3.3.2. Clinical study

Fig. 5. The FOV of the TEE is a pyramid shape structure which can be defined from
any TEE volume. Ten image planes were selected within the TEE FOV which are
parallel to the transducer of the TEE probe. The interval between two image planes
was 10 mm. The landmarks defined on the four corners of the image planes were
used to examine the potential error for targets at different depths caused purely by
the error of the 2D–3D registration algorithm.
To assess the accuracy of the clinical studies, we identify the
interventional devices (ablation catheters, trans-catheter valves)
in the TEE volume and extracted their center lines manually. The
center lines were then projected onto the X-ray images using the
TEE to X-ray registration algorithm described above. A number of
points were then defined along the center line of the same inter-
ventional device on the X-ray image. The registration error in term
of mPD was calculated using the mean distance between the points
and the projected center line of the interventional devices. Point-
to-line distances cannot capture the errors that are tangential to
44 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

of the capture range analysis using a randomly selected clinical

Fig. 7. An example of a successful 2D–3D image registration. The X-ray images
(first row) were acquired during a TAVI case from RAO 29° (left) and LAO 30° (right).
The nano-CT volume was registered with and projected onto both of the images
(second row).
dataset. Given the success rate threshold of 2.5 mm, the capture
range of the 2D–3D registration algorithm was 9.2 mm (Fig. 8a).
The error generated by most of the failed registrations (89.2%)
was greater than 10 mm while most of the successful registrations
created an error of less than 1 mm (98.7%). The success rate thresh-
old can be altered between 1 mm and 10 mm without considerably
changing the capture range result. This is in-line with the previous
findings in (van de Kraats et al., 2005). Fig. 8b shows the potential
TEE and X-ray registration error propagated from the 2D–3D regis-
tration at different depths of the ultrasound cone for all the suc-
cessful 2D–3D registrations. In practice, the imaging targets such
as the heart valves were routinely positioned in the middle of
the ultrasound FOV at around 5–6 cm depth. The potential mTRE
caused by the 2D–3D registration at 6 cm depth is approximately
1 mm. Overall more than 95% of the registrations achieved an
mTRE of less than 2.5 mm. The capture range study was carried
out for each of the clinical data sets. The results are summarized
in Table 1. For all the successful registrations (mTRE 6 2.5 mm),
the mean error in terms of rotations and translations are listed in
Table 2. Fig. 9 shows the stating positions for all the failed registra-
tions in terms of rotations and translations. It suggests only 4.2% of
the failed registrations start within a region of 10 mm  10°. How-
ever, if the starting position of the TEE probe is within a region of
12 mm  12°, the percentage increases dramatically to 31.6%. This
indicates that an initial approximate registration should correct

the line. However, several of the devices, especially the EP cathe-

ters, had substantial curvature, reducing the significance of this Table 1
problem. The capture range of the 2D–3D registration. The testing data were patient data
collected during the clinical procedures.
4. Experimental results Study Patient Procedure Image size Capture range (mm)
ID
Two X-ray One X-ray
4.1. Nano-CT and X-ray image registration images image
1 1 EP 1024  1024 9.7 10.4
In this section, we present the image registration results be- 2 2 EP 1024  1024 11.4 11.2
tween the nano-CT volume of the TEE probe and the X-ray images. 3 2 EP 1024  1024 10.3 9.8
4 3 TAVI 512  512 10.7 10.1
5 3 TAVI 512  512 9.2 8.7
4.1.1. Capture range and initial registration
6 4 TAVI 512  512 9.6 9.4
Fig. 7 shows an example of a successful registration between 7 5 TAVI 512  512 8.7 9.8
the nano-CT volume and two X-ray images. Fig. 8 shows the results

Fig. 8. (a) For each of the datasets, 1400 starting positions with mTRE ranging evenly from 1 mm to 15 mm were used to examine the capture range of the intensity-based
2D–3D image registration algorithm. (b) For all the successful registrations, landmarks were defined within the TEE FOV at different depths to examine the potential target
error caused by the error of the 2D–3D image registration. Presuming that the 2D–3D image registration was the sole error source, the red curve shows the success rate of the
TEE and X-ray image registration for the targets defined within the TEE FOV at different depths.
 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49 45

Table 2
The errors of all the successful registration in terms of rotation and translation.

Translation Rotation
tx (mm) ty (mm) tz (mm) rx (°) ry (°) rz (°)
Mean err. 0.06 0.16 0.04 1.16 0.17 1.05
Std. dev. 0.03 0.06 0.03 0.23 0.11 0.58
Max. err. 0.47 1.07 1.06 3.13 1.47 2.46

Fig. 10. (a) Comparison of 3D errors in term of mTRE for the 2D–3D registration
Fig. 9. The starting positions of all the failed 2D–3D registrations in terms of
using bi-plane X-ray and mono-plane X-ray. The testing data were acquired from
translation and rotation. Only 4.2% of the failed registrations started from within a
the first of the TAVI procedures reported in this paper. Potentially, the 2D–3D
region of 10 mm  10°.
registration using one X-ray image could produce considerable error in the X-ray
projection direction, reducing the overall success rate. (b) The majority of the error
is in the direction of the X-ray projection, the in-plane error mPD.is 1.8 ± 1.3 mm
the misalignment to within a 10 mm  10° region to be subse- compared to the overall 3D error of 6.8 ± 7.8 mm.
quently successful for automatic registration.
Technically, it is possible to register the nano-CT volume with
mono-plane X-ray images. The capture range of the image registra-
jittering motion which clearly reveal the pattern of cardiac and
tion using one X-ray image is similar to those using bi-plane X-ray
respiratory motion. For both cases, the jittering motion introduces
images. However, it could introduce a considerable error in term of
a maximum error of around 4 mm in the probe localization
mTRE to the target defined within the FOV of TEE (Fig. 10a). The
(4.1 mm and 3.6 mm, respectively). The maximum errors contrib-
majority of the error occurs along the direction of the X-ray beam
uted by the rotational components of the jittering motion are
and does not impact considerably if the echo data is used as an
1.6 mm and 1.0 mm, respectively. Although our experiment shows
overlay image on top of the X-ray fluoroscopy as demonstrated la-
the majority of the jittering motion is translational, the error can
ter in Figs. 13 and 14. Fig. 10b shows that the average in-plane reg-
still make a considerable impact on the accuracy of the overall
istration error in terms of mPD is 1.8 ± 1.3 mm. We do not restrict
TEE and X-ray registration especially if the target object locates
the number of X-ray images used in our clinical workflow. The
at the far end of the echo FOV. In the proposed clinical workflow,
decision will be made based on the requirement of the interven-
a real-time TEE probe tracking algorithm will minimize the error
tional procedures. More details will be discussed in Section 5.
caused by the jittering motion. Before such an algorithm becomes
available, physicians will need to accommodate the registration er-
4.1.2. Registration speed ror using the motion pattern of the interventional devices observed
The computational time for the 2D–3D image registration de- in the X-ray fluoroscopy and in the echo volume.
pends on several factors including the starting position of the
nano-CT volume, the size of the X-ray images and the number of
the X-ray images used in the registration. From over 30,000 regis-
4.2. TEE probe calibration
trations performed using different starting positions, the computa-
tional time of our 2D–3D image registration was recorded and is
The residual calibration error from Eq. (3) was 2.0 mm. By using
summarized in Table 3.
the calibration matrix, the last two volumes that were not involved
in the calibration were transformed to the C-arm space as where
4.1.3. Jittering motion error the landmarks from the X-ray images. The mTREs computed in
X-ray images acquired from a TAVI procedure (Fig. 11a) and an C-arm space were 4.6 ± 1.1 mm and 5.0 ± 0.8 mm. The determina-
EP procedure (Fig. 11b) were used to assess the error caused by the tion of the landmark positions from the TEE volume was difficult
jittering motion of the TEE probe. Fig. 11 shows the curves of the due to the noise and the shadowing effects of ultrasound. The
46 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49
Fig. 11. The TEE probe localization error caused by the jittering motion. X-ray images were acquired during (a) a TAVI procedure and (b) a cardiac EP procedure.
mTREs show not only the registration error but also the subjective
error in the determination of the landmark positions.
4.3. Phantom study
The 3D positions of the landmarks were reconstructed in C-arm
space using two X-ray images acquired from LAP 45° and RAO 45°.
The mean reconstruction error in term of mPD was much less than
1 mm (0.44 mm using the image acquired from LAO 45° and
0.14 mm using image from RAO 45°). The mean landmark registra-
tion error for the MR to C-arm registration was 1.03 mm. The
point-to-surface distance errors between TEE and MRI data are
summarized in Table 4. The mean TEE to C-arm registration error
was in a range 1.5–2.4 mm. Fig. 12 shows the MRI-derived LV
and the RV surfaces registered with two TEE volumes in C-arm
space.
4.4. Clinical studies
Fig. 13 shows the registration results when different catheters
were used for two patients undergoing EP procedures. At this
Fig. 12. Two TEE images and the MR volume of the heart phantom are shown
transformed to C-arm space using either the proposed method or the landmark
registration, respectively. Presuming a comparatively high accuracy for the
landmark registration, the alignment between the MR volume and the TEE images
indicates the accuracy of the proposed method.
stage, the data acquisition of TEE and X-ray data was not synchro-
nized. The end systolic phase was manually selected from the
X-ray and TEE sequences to calculate the mPD shown in Table 5.
However, the catheter motion (caused by the cardiac motion
including cardiac contraction and respiratory motion) shown in
the X-ray images was more considerable than that shown in the
TEE volume. This is because the catheter motion shown in TEE
was partly canceled by the motion of the TEE probe itself. At pres-
ent, our 2D–3D registration algorithm is not fast enough to com-
pensate for the TEE probe motion in real-time. This error will be
additive in the registration results.
Fig. 14 shows the X-ray images and a TEE volume acquired in one
of the TAVI procedures. The diameter of the TAVI catheter (18-22F or
6–7 mm) is considerably larger than the EP catheters (8F, 3 mm).
Although both the TAVI catheters and the valves can be easily iden-
tified from the TEE volume, it is difficult to determine the exact loca-
tion of the replacement valve in the TEE volume. Fig. 14a shows the
position of the replacement valve was clearly visible in the X-ray
images. The combined image of X-ray and TEE reveals that the
replacement valve had passed through the aortic valve into the left
ventricle. Without seeing the overlay image, the physician decided
to deploy the replacement valve in this sub-optimal position. Post-
deployment Doppler revealed severe paravalvular leakage and poor
hemodynamic performance. The motion of the mitral valve was also
obstructed with the mis-placed aortic valve. A second replacement
valve was inserted to stop the leakage. This case highlights the po-
tential clinical impact of our technique.
The registration accuracy for all the clinical cases (two EP and
three TAVI) is summarized in Table 5.
5. Discussion
5.1. Robustness and accuracy
The novelty of our method is that we employed an image-based
2D–3D registration algorithm to localize the TEE probe. The perfor-
mance of 2D–3D registration algorithms is usually data dependent.
However, our task is more straightforward than many previously
reported studies. Firstly, the source object in our study (the TEE
probe) is always the same. Secondly, the visibility and contrast of
the TEE probe in the X-ray images is relatively constant and is un-
likely to be affected by other objects because the density of the TEE
 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49
Fig. 13. TEE volumes were registered with and overlaid onto the X-ray images for
two patients undergoing EP procedures. The original X-ray images are shown in the
first row. For the first patient (first column), the TEE volume shows a deca-polar
catheter positioned in right atrium. The dots highlight the position of the deca-polar
catheter in the background X-ray image. For the second patient (second column), a
lasso catheter was inserted into the left atrium. For both patient data, the
registration was successful as the catheters in the TEE volumes showed a good
alignment with the background X-ray images.
probe is considerably higher than the soft tissues and the spine.
The robustness study described in Section 3 showed consistent
registration results when the start positions were within the cap-
ture range of the 2D–3D registration algorithm. Our study showed
the capture range of the 2D–3D registration was approximately
10 mm in terms of mTRE. An additional examination of all the
failed registrations found that most of the failed registrations
started outside the region of 10°  10 mm in terms of the magni-
tudes of rotations and translations, suggesting that an initial man-
ual alignment was required to bring the nano-CT volume into the
Fig. 14. Co-registered echo and X-ray images for one of the TAVI patients. (a) The X-ray image shows the interventional devices including the catheter, the guide wire and the
replacement valve; (b) the 3D TEE reveals cardiac anatomy including the aorta and the valves as well as the catheter. However, it was difficult to identify the exact location of
the replacement valve; (c) the highlighted area in the combined image of TEE and X-ray shows the replacement valve had passed through the aortic valve into the left
ventricle.
47
Table 3
The average computation time of the 2D–3D registration.
Num. of X-ray Image size Num. of Mean time SD
images registrations (s) (s)
1 1024  1024 8400 8.0 2.1
2 1024  1024 4200 11.3 4.1
1 512  512 12,800 1.8 0.6
2 512  512 6400 5.4 1.6
capture range before the automated registration would be success-
ful. However, the end point of the proposed method is to register
the TEE images with the X-ray images.
The accuracy of our TEE probe calibration did not compare
favorably with the sub-millimeter results obtained from classically
tracked ultrasound probes (Mercier et al., 2005). We repeated our
calibration to check for any mistakes that may have been made
during the calibration process but the results were very similar
to our initial experiment. Looking at Eq. (3), we see that there
are three sources of error. Firstly, there is the error of locating fidu-
cials in the US image space; secondly, there is the error of locating
fiducials in the 2D X-ray image space and back-projecting these to
3D C-arm space; and finally, there is the error of performing the
2D–3D registration. For classically tracked US probes, the tracking
error will be in the order of 1–2 mm, in the case of EM tracking, or
less, in the case of optical tracking. For tracking using 2D–3D reg-
istration, we envisage the errors to be greater and furthermore we
must include the error of the back-projection, which is typically
1 mm. Therefore, the reported results of a 2 mm residual error
from Eq. (3) and 3D mTRE of 4.6/5 mm are expected from the cur-
rent methodology. There could be scope to improve this substan-
tially using a multi-crosswire phantom coupled to a large
number of ultrasound and X-ray image acquisitions for calibration.
The X-ray and TEE image registration results of both the phan-
tom and clinical studies are encouraging in terms of the target reg-
istration accuracy achieved. The clinical accuracy requirement is
influenced by the type of clinical procedure (Linte et al., 2010).
The values of less than 3 mm achieved for the heart phantom and
1.5–4 mm achieved for the clinical cases are within the accuracy
requirement for navigating within the cardiac chambers and great
vessels that have a minimum diameter of approximately 10 mm
in adults. However, in this study, as for many other clinical studies,
it was difficult to find a gold standard. mPD can be used as an indi-
cator for the accuracy but for the TAVI cases, the interventional
48 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49

Table 4 tem, although neither of the two studies were performed in the clin-
The point-to-surface error represents the error in the registration of TEE and MRI in C- ical environment. At present, there is no commercial EM tracker-
arm space. For LV and RV, the points were defined on all the six TEE volumes, 10 for
each volume. For LA, RA and the valves, points were defined on the volumes where
embedded TEE probe available. To use an EM tracking system with
the borders were clearly visible. The valves were not segmented from the MR volume. the TEE probe, the TEE probe must be modified so that an EM sensor
The point-to-surface distance for the valves was the distance of a point defined on the can be fitted within it. Compared to the use of an EM tracker, the first
valves to the top surface of the LV or the RV. advantage of our method is that it does not use any additional track-
Num. of points Mean error (mm) SD (mm) Max. error (mm) ing device. The accuracy of the EM tracking system relies on a static
LV 66 2.0 1.5 4.7
metal environment. With the presence of the moving X-ray C-arm
RV 60 1.9 1.2 4.9 and other medical devices including the TEE probe itself, the reli-
Valves 16 1.9 1.3 2.7 ability of the EM tracking system is questionable (Hastenteufel et
LA 22 1.5 0.8 2.8 al., 2006). Additionally, the methodology presented for the co-regis-
RA 16 2.0 1.4 4.4
tration of 3D TEE and X-ray fluoroscopy data is better suited for the
LVPM 20 2.4 1.8 3.2
Overall 200 1.9 1.2 4.9 routine clinical work flow.

5.3. Mono-plane X-ray and bi-plane X-ray

Table 5
Registration error results from the clinical cases. Devices were projected from the TEE
volume data to the X-ray image data using the computed 2D–3D registration. The
mPD was computed for each device.
Patient ID Procedure mPD (mm)
1 EP 3.1
2 EP 1.5
2 EP 3.2
3 TAVI 2.7
4 TAVI 4.2
5 TAVI 2.3
devices are relatively straight in the FOV of the TEE images and
therefore the tangential error along the interventional devices can-
not be captured. However, the validity of mPD as an accuracy met-
ric was supported in the unsuccessful TAVI case, where the clinical
outcome matches the finding of the TEE and X-ray registration. The
accuracy of the proposed registration method is also confirmed in
one of the EP cases where a circular catheter was used as the refer-
ence for mPD calculation.
At this stage, all the clinical data we used were processed off-
line. The X-ray images and the TEE volumes were not acquired at
the same time. Although we attempted to minimize the registra-
tion error by using the images acquired in the same cardiac and
respiratory phases, the error caused in the cardiac phase estima-
tion will be propagated to the end registration result. Such error
is likely to be further reduced in the future when real-time X-ray
and ultrasound data can be acquired simultaneously.
The jittering motion of the TEE probe during a clinical procedure
is another source of error which potentially could affect the accu-
racy of the TEE and X-ray registration. Experimental results show
the error introduced by the jittering motion can be as much as a
4 mm error in the probe localization. Fig. 8a suggests this error
can be easily removed using the 2D–3D registration. The success
rate of the 2D–3D image registration is 100% given the starting po-
sition of the source volume is within 5 mm capture range. However,
the speed of the current 2D–3D registration algorithm needs to be
improved considerably to satisfy the requirement of real-time
tracking.
5.2. Comparison with EM tracking systems
At present, our 2D–3D registration algorithm takes around 5 s to
determine the 3D position of the TEE probe using two 512  512 X-
ray images. EM tracking systems determine the location of objects
that are embedded with the EM sensors in real-time. More impor-
tantly, it is not required that the EM sensor be in the line-of-sight
of the magnetic field generator, as for optical tracking systems. Pre-
vious studies (Gao et al., 2009; Jain et al., 2009) confirmed that it
was possible to localize the TEE probe by using an EM tracking sys-
The 2D–3D registration algorithm is not sensitive to detecting
misalignment along the direction of the X-ray beam so that the
2D–3D registration algorithm using mono-plane images often
SD (mm)
produces considerable error in the direction of projection.
Introducing one more X-ray image acquired from a different angle
2.6
effectively solves this problem, allowing an accurate registration in
1.6
2.5 all six DOFs although the radiation dose will be double (but very
1.8 small when compared to entire procedure dose). This can be
3.0 performed using bi-plane X-ray systems but these are not com-
2.1
monly available. A more conventional method is to use sequential
bi-plane images from a mono-plane X-ray system. However, with
sequential bi-plane acquisition, correct phase-matching (for both
respiratory phase and cardiac phase) of the X-ray data must be
carried out.
Clinically, the 2D–3D registration with bi-plane X-ray is not al-
ways necessary. The clinical workflow includes two different ways
to visualize and utilize the registration results, Echo ? X-ray pro-
jection and X-ray ? Echo projection. Echo ? X-ray projection
overlays the echo volume on top of the X-ray images producing a
‘roadmap’ to navigate the clinical procedures such as atrial fibrilla-
tion (AF) ablation and TAVI. In these cases, the 2D–3D registration
using mono-plane X-ray is sufficient as the accuracy in the direc-
tion of projection is not a major concern. However, in cases which
the 3D position of the interventional devices must be precisely
localized, bi-plane X-ray images will be needed to ensure a precise
TEE and X-ray registration in all six degrees of freedom. For exam-
ple, in some AF ablation cases, the locations of the ablation sites
need to be recorded for reviewing purposes. An example for the
application of X-ray ? Echo projection could be: (1) determine
the location of the ablation catheter tip from the X-ray image;
(2) back-project the 2D location of the catheter tip to C-arm space,
producing a line intersecting the registered TEE volume; and (3)
determine the 3D position of the ablation catheter tip either man-
ually or automatically from the TEE volume.
5.4. Limitations and future work
One of the main objectives of this study was to assess the clinical
feasibility of the proposed method. Although all the clinical data
were acquired specifically for this study, the data processing was
done off-line. However, the transition to real-time or near-real-
time capability is relatively straightforward in this case, with all
the required components already present, including live data
streaming from both X-ray and ultrasound systems. Our future
work will focus on the implementation of the live functionality
and also improvements in the visualization of the co-registered
data. The 3D TEE images were acquired in full volume mode. Our
TEE probe was calibrated using full volume echo images. The clini-
cal procedures are usually guided by live 3D echo images whose
FOV is about a quarter of the full volume images. Therefore,
 G. Gao et al. / Medical Image Analysis 16 (2012) 38–49
co-registration with a global roadmap from either CT or MR image
data could assist in clarifying the anatomical context of the small
FOV echo data.
6. Conclusions
In this study, we have described a novel TEE to X-ray fluoros-
copy registration technique. The method was successfully evalu-
ated using phantom data and five clinical datasets acquired from
two different types of cardiac interventional procedures, including
cardiac EP procedures and TAVI procedures. The experimental re-
sults show that our method is fast, robust and accurate. It is likely
that such a co-registration and visualization technology is going to
have a significant impact in the field of image-guided cardiac
interventions.
References
Barker, G.H., Shirali, G., Ringewald, J.M., Hsia, T.Y., Bandlsode, V., 2008. Usefulness of
live three-dimensional transesophageal echocardiography in a congenital heart
disease center. The American Journal of Cardiology 103, 1025–1028.
Brost, A., Liao, R., Hornegger, J., and Strobel, N., 2010. 3-D Respiratory Motion
Compensation during EP Procedures by Image-Based 3-D Lasso Catheter Model
Generation and Tracking. Medical Image Computing and Computer Assisted
Intervention (MICCAI’09) Medical Image Computing and Computer Assisted
Intervention (MICCAI’09), pp. 394–401.
De Buck, S., Meas, F., Ector, J., Bogaert, J., Dymarkowski, S., Heidbuchel, H., Suetens,
P., 2005. An augmented reality system for patient-specific guidance of cardiac
ablation procedures. IEEE Transactions on Medical Imaging 24, 1512–1524.
Gao, G., Penney, G., Gogin, N., Cathier, P., Arujuna, A., Wright, M., Caulfield, D.,
Rinaldi, A., Razavi R., Rhode, K.S., 2010. Rapid Image Registration of Three-
dimensional Transesophageal Echocardiography and X-ray Fluoroscopy for the
Guidance of Cardiac Interventions. Information Processing in Computer-
Assisted Intervention (IPCAI’10) Information Processing in Computer-Assisted
Intervention (IPCAI’10).
Gao, G., Reddy, K., Ma, Y., Rhode, K.S., 2009. Real-time Compounding of Three-
Dimensional Transesophageal Echocardiographic Volumes: The Phantom Study.
IEEE Engineering in Medicine and Biology Society (EMBC’09) IEEE Engineering
in Medicine and Biology Society (EMBC’09), pp. 499–502.
Hastenteufel, M., Vetter, M., Meinzer, H.P., Wolf, I., 2006. Effect of 3D ultrasound
probes on the accuracy of electromagnetic tracking systems. Ultrasound in
Medicine and Biology 32, 1359–1368.
Hawkes, D.J., Mol, C.B., Colchester, A.C.F., 1987. The accurate 3-D reconstruction of
the geometric configuration of vascular trees from X-ray recordings. Physics
and Engineering of Medical Imaging 250, 258.
Hipwell, J.H., Penney, G., Mclaughlin, R.A., Rhode, K.S., Summers, P., Cox, T.C., Byrne,
J.V., Noble, J.A., Hawkes, D.J., 2003. Intensity based 2D–3D registration of
cerebral angiograms. IEEE Transactions on Medical Imaging 22, 1417–1426.
Huang, X.S., Ren, J., Guiraudon, G., Boughner, D., Peters, T., 2009. Rapid dynamic
image registration of the beating heart for diagnosis and surgical navigator. IEEE
Transactions on Medical Imaging 28, 1802–1814.
Jain, A., Gutierrez, L., Stanton, D., 2009. 3D TEE Registration with X-ray Fluoroscopy
for Interventional Cardiac Applications. Functional Imaging and Modeling of the
Heart (FIMH’09) Functional Imaging and Modeling of the Heart (FIMH’09), pp.
321–329.
King, A.P., Boubertakh, R., Rhode, K.S., Ma, Y., Chinchapatnam, P., Gao, G., Tangcharoen,
T., Ginks, M., Cooklin, M., Gill, J.S., Hawkes, D.J., Razavi, R., Schaeffter, T., 2009. A
subject-specific technique for respiratory motion correction in image-guided
cardiac catheterization procedures. Medical Image Analysis 13, 419–431.
King, A.P., Rhode, K.S., Ma, Y., Yao, C., Jansen, C., Razavi, R., Penney, G., 2010.
Registering preprocedure volumetric images with intraprocedure 3-D
ultrasound using an ultrasound imaging model. IEEE Transactions on Medical
Imaging 29, 924–937.
Klemm, H., Steven, D., Johnsen, C., Ventura, R., Rostock, T., Lutomsky, B., Risius, T.,
Meinertz, T., Willems, S., 2007. Catheter motion during atrial ablation due to the
beating heart and respiration: impact on accuracy and spatial referencing in
three-dimensional mapping. Heart Rhythm 4, 587–592.
Knecht, S., Wright, M., Akrivakis, S., Nault, I., Matsuo, S., Chaudhry, G.M., Haffajee, C.,
Sacher, F., Lellouche, N., Miyazaki, S., Forclaz, A., Jadidi, A.S., Hocini, M., Ritter, P.,
49
Clementy, Haissaguerre, M., Orlov, M., Jais, P., 2010. Prospective randomized
comparison between the conventional electroanatomical system and three-
dimensional rotational angiography during catheter ablation for atrial
fibrillation. Heart Rhythm 7, 459–465.
Kovoor, P., Ricciardello, M., Collins, L., Uther, J., Ross, D., 1998. Risk to patients from
radiation associated with radiofrequency ablation for supraventricular
tachycardia. Circulation 98, 1534–1540.
Linte, C.A., Moore, J., Peters, T.M., 2010. How Accurate is Accurate Enough? A Brief
Overview on Accuracy Considerations in Image-Guided Cardiac Interventions.
In: Proceedings of 32nd Annual International Conference of the IEEE EMBS, pp.
2313–2316.
Mackensen, G.B., Hegland, D., Rivera, D., Adams, D.B., Bahnson, T.D., 2008. Real-time
3-dimensional transesophageal echocardiography during left atrial radio
frequency catheter ablation for atrial fibrillation. Circulation: Cardiovascular
Imaging 1, 85–86.
Manzke, R., Meyer, C., Ecabert, O., Peters, J., Noordhoek, N., Thiagalingam, A., Reddy,
V., Chan, C.C., Weese, J., 2010. Automatic segmentation of rotational X-ray
images for anatomic intra-procedural surface generation in atrial fibrillation
ablation procedures. IEEE Transactions on Medical Imaging 29, 260–272.
Mercier, L., Langø, T., Lindseth, F., Collins, D.L., 2005. A review of calibration
techniques for freehand 3-D ultrasound systems. Ultrasound in Medicine &
Biology 31, 449–471.
Modan, B., Keinan, L., Blumsterin, T., Sadetzki, S., 2000. Cancer following cardiac
catheterization in childhood. International Journal of Epidemiology 29, 424–
428.
Penney, G., Blackalll, J.M., Hamady, M.S., Sabharwal, T., Adam, A., Hawkes, D.J., 2004.
Registration of freehand 3D ultrasound and magnetic resonance liver images.
Medical Image Analysis 8, 81–91.
Penney, G., Weese, J., Little, J.A., Desmedt, P., Hill, D.L., Hawkes, D.J., 1998. A
comparison of similarity measures for use in 2-D–3-D medical image
registration. IEEE Transactions on Medical Imaging 17, 586–595.
Razavi, R., Hill, D.L., Keevil, S.F., Miquel, M.E., Muthurangu, V., Hegde, S., Rhode, K.S.,
Barnett, M., van Vaals, J., Hawkes, D.J., Barker, E., 2003. Cardiac catheterization
guided by MRI in children and adults with congenital heart disease. The Lancet
362, 1877–1882.
Rhode, K.S., Hill, D.L., Edwards, P.J., Hipwell, J.H., Rueckert, D., Sanchez-Ortiz, G.,
Hegde, S., Rahunathan, V., Razavi, R., 2003. Registration and tracking to
integrate X-ray and MR images in an XMR facility. IEEE Transaction on
Medical Imaging 22, 1369–1378.
Rhode, K.S., Sermesant, M., Brogan, D., Hegde, S., Hipwell, J.H., Lambiase, P.,
Rosenthal, E., Bucknall, C., Qureshi, S.A., Gill, J.S., Razavi, R., Hill, D.L., 2005. A
system for real-time XMR guided cardiovascular intervention. IEEE Transaction
on Medical Imaging 24, 1428–1440.
Roche, A., Pennec, X., Malandain, G., Ayache, N., 2001. Rigid registration of freehand
3D ultrasound with MR images: a new approach combing intensity and
gradient information. IEEE Transactions on Medical Imaging 20, 1038–1049.
Shekhar, R., Zagrodsky, V., 2002. Mutual information-based rigid and non-rigid
registration of ultrasound volumes. IEEE Transactions on Medical Imaging 21,
9–22.
Silverstry, F.E., Kerber, R.E., Brook, M.M., Caroll, J.D., Eberman, K.M., Goldstein, S.A.,
Herrmann, H.C., Homma, S., Mehran, R., Packer, D.L., Parisi, A.F., Pulerwitz, T.,
Seward, J.B., Tsang, T.S., Wood, M.A., 2009. Echocardiography-guided
interventions. Journal of American Society of Echocardiography 22, 213–231.
Sra, J., Narayan, G., Krum, D., Malloy, A., Cooley, R., Bhatia, A., Dhala, A., Blanck, Z.,
Nangia, V., Akhtar, M., 2007. Computed tomography–fluoroscopy image
integration-guided catheter ablation of atrial fibrillation. Electrophysiology
18, 409–414.
Turgeon, G.-A., Lehman, G., Guiraudon, G., Drangova, M., Holdsworth, D., Peters, T.,
2005. 2D–3D registration of coronary angiograms for cardiac procedure
planning and guidance. Medical Physics 32, 3737–3749.
van de Kraats, E., Penney, G., Tomazevic, D., van Walsum, T., Niessen, W., 2005.
Standardized evaluation methodology for 2-D–3-D registration. IEEE
Transactions on Medical Imaging 24, 1177–1189.
Wein, W., Carnus, E., John, M., Diallo, M., Duong, C., Al-Ahmad, A., Fahrig, R.,
Khamene, A., Xu, C., 2009. Towards guidance of electrophysiological procedures
with real-time 3D intracardiac echocardiography fusion to C-arm CT. Medical
Image Computing and Computer Assisted Intervention (MICCAI’09) Medical
Image Computing and Computer Assisted Intervention (MICCAI’09), pp. 9–16.
Yu, H., Fahrig, R., Pelc, N.J., 2005. Co-registration of X-ray and MR fields of view in a
hybrid XMR system. Journal of Magnetic Resonance Imaging 22, 291–301.
Yushkevich, P., Piven, J., Hazlett, H., Smith, R., Ho, S., Gee, J., Gerig, G., 2006. User-
guided 3D active contour segmentation of anatomical structures: significantly
improved efficiency and reliability. Neuroimage 31, 1116–1128.