You are on page 1of 22


AR in the
Hybrid Opera6ng Room
Fernando Barrera
ICube, University of Strasbourg, CNRS, IHU Strasbourg, France

Research Group CAMMA

Computa(onal Analysis and Modeling of Medical Ac(vi(es
AR in the OR

Medical prac11oners are confronted with more and more

data and informa1on during a procedure:
Imaging data, pa1ent informa1on, surgical instrument tracking
AR permits in-situ visualiza1on of key informa1on related to
the performance of the procedure

Linte et al., On mixed reality environments for minimally

Jolesz et al., Image-guided procedures invasive therapy guidance: Systems architecture,
and the OR of the future, RSNA 1996 successes and challenges in their implementa(on from
laboratory to clinic, Comput Med Imaging Graph 2013
AR in the OR

Towards context-aware radia1on protec1on

for the hybrid opera1ng room

Loy Rodas and Padoy, 3D Global Es(ma(on and Augmented Reality Visualiza(on of Intra-opera(ve X-ray Dose, MICCAI 2014
Mobile AR in the OR

Clinical applica1on: display 3D informa1on related to

radia1on safety directly in the clinicians view
In-situ feedback about an upcoming X-ray image:
Robustness to mo1on in the scene and occlusions
Impossibility of having mul1ple markers visible from all
X-ray device parameters
interes1ng viewpoints
Pa1ent radia1on exposure
Pre-computed as in [1]
Modeling of the scene beyond the pa1ent is required
3D distribu1on of radia1on

[1] Loy Rodas and Padoy, 3D Global Es(ma(on and Augmented Reality Visualiza(on of Intra-opera(ve X-ray Dose, MICCAI 2014
Related work

AR relies on an accurate registra1on of the users point of view

and tracking of the viewing device, currently achieved by:

Tracking markers [1] Registering a mesh to a model [2]

Markers can be intrusive and Changes in the posi(ons of equipment
interfere with the procedure at or clinicians can invalidate any a priori
hand models

[1] Navab et al., First Deployments of Augmented Reality in Opera(ng Rooms, Computer 2012
[2] Kilgus et al., Mobile Markerless augmented reality and its applica(ons in forensic medicine, IJCARS 2015

Mul6-view RGBD camera system

2 RGBD cameras rigidly mounted to the ceiling and 1 RGBD camera xed to a
registered to a room global coordinate system hand-held screen

Equipment detec6on for tracking ini6aliza6on / camera relocaliza6on

Simultaneous equipment detec1on in at

least 2 views

Natural markers for obtaining an ini1al

registra1on of mobile camera to room

Detec1on on a sta1c camera Detec1on on moving camera


Frame-to-frame moving camera tracking with KinectFusion-like* approach

New equipment detec1ons allows fast

relocaliza1on when tracking is lost

TSDF volume Moving device trajectory

* Newcombe et al., Kinect-Fusion:

Real-(me dense surface mapping
and tracking, ISMAR 2011
Equipment detec1on

Mul6modal template matching approach

based on LINEMOD*
1. Iden1fy ongoing X-ray projec1on via equipment detec1on
2. Equipment and moving camera in constant mo1on
3. Equipment not necessarily placed over a planar surface
4. Large equipment size: not fully visible and subject to occlusions
*Hinterstoisser et al., Model Based Training, Detec(on and Pose Es(ma(on of Texture-Less 3D Objects in Heavily CluVered Scenes, ACCV 2012
Equipment detec1on

Template genera6on

Genera1on of virtual RGBD images to cover for

Camera viewpoints recursively sampled
Extrac1on of color gradients and
each possible camera viewpoint
around the model
surface normals from each pair
Equipment detec1on

Needed for clinical applica6on:

Detec1on of the current X-ray device congura1on
3D pose es1ma1on of moving device

Not possible using a single camera: ambigui1es when equipment and camera move

Equipment detec1on also needed in sta1c cameras

Equipment detec1on

Needed for clinical applica6on:

Detec1on of the current X-ray device congura1on
3D pose es1ma1on of moving device

Template genera1on procedure is repeated for a set of 3D orienta1ons

Larger database of templates

Equipment detec1on

A priori informa6on used to lter improbable views

Template database reduced by 70% by keeping only the relevant templates

Moving camera possible

Moving devices
poten1al areas of mo1on
for our clinical applica1on
Equipment detec1on

Part-based detec6on

Equipment detected even when it is not fully visible

Equipment detec1on

Dynamic template subset selec6on

Sta1c camera
Detec1on of devices congura1on
in sta1c camera

Search space is reduced:

faster tracking ini6aliza6on
Qualita1ve results

Mobile Markerless AR system for radia1on awareness

Mul1 RGBD camera system for equipment detec1on in the OR

Non-intrusive approach: natural markers for camera localiza1on

Dynamic selec1on of templates subsets + detec1on of parts permits
to reduce the number of tested templates in the moving view
New applica1on for mobile AR in the OR

Radia1on maps not computed in real-1me & pa1ent not tracked
Tracking accuracy aected by errors from:
Mul1-camera system calibra1on
Quality of depth map from RGBD cameras (Asus X1on Pro)
Quality of the equipment detec1on

Fully context-aware radia6on awareness system with

full 3D OR understanding
Body-part dose es6ma6on

* Kadkhodamohammadi et al., Pictorial Structures on RGB-D images for Human

Pose Es(ma(on in the Opera(ng Room, MICCAI 2015
Thank you !


Research Group CAMMA , ICube, University of Strasbourg


Preliminary quan1ta1ve evalua1on

Ground-truth moving camera pose obtained with an op1cal tracking system

X-ray device
0 90 180 270 Mean std
ICP error (mm) 210.9 81.0 222.8 79.8 162.7 51.8 234.9 91.7 207.8 76.0
Rx error 5.1 2.5 3.6 2.7 3.9 1.4 5.3 3.3 4.4 2.4
Ry error 5.1 2.2 10.1 2.0 4.4 1.2 6.7 2.8 6.5 2.0
Rz error 3.4 1.4 9.2 0.5 5.3 1.3 4.2 1.0 5.6 2.5
Mean R error 4.5 1.8 7.6 1.6 4.5 1.0 5.4 2.2 5.5 1.6
T error (mm) 88.2 51.5 103.8 59.2 119.2 25.1 97.8 42.8 100.2 44.6
Moving camera tracking error

Radia1on propaga1on simula1on framework

C-arm congura6on and room lay-out Irradia6on parameters
modeling obtained from X-ray device
Tube tension
Air Kerma

Simula1on of X-ray Energy spectra

Calibra6on with
Simula6on of Radia6on Propaga6on (Geant4) test dosimeters
X-ray tube: source of photons with energies sampled from
simulated spectrum

Modeling of Compton Scaoering, Rayleigh Scaoering and

Photoelectric Eect Correc6on factor

Intra-opera6ve X-ray Dose values

Loy Rodas and Padoy, 3D Global Es(ma(on and Augmented Reality Visualiza(on of Intra-opera(ve X-ray Dose, MICCAI 2014
Radia1on propaga1on simula1on framework

Simula1on of X-ray dose in all room enables to es1mate the deposited

dose over the pa1ents surface

Using a scanned model of a phantom:
We iden1ty the voxel each surface point intersects
We color each point according to the dose in the intersec1on voxel

AR Visualiza1on of pa1ents surface dose for dierent device


X-ray device projec1on 0 90 180 225