Augmented Reality: Integrating Computer Graphics With Computer Vision Mihran Tuceryan

Augmented Reality
Integrating Computer Graphics with

Computer Vision
Mihran Tuceryan
August 16, 1998 ICPR 98 1

Definition
XCombines real and virtual worlds and
objects
XIt is interactive and real-time
XThe interaction and alignment of real and
virtual objects happens in 3D (I.e., not 2D
overlays).

An Example Application
Electric wires shown

as “augmentation” in a
physical room.

Motivation
Claim:
Augmented Reality can be deployed in
much more useful applications in the real
world than virtual reality.

Possible Applications
Maintenance and
Repair
Ref: ECRC
X Architecture and Construction
Before After
Images: courtesy of CICC project at ZGDV, Fraunhofer IGD
X Interior Design
Ref: ECRC
X Aid in construction sites: X-ray view of a wall
Image: courtesy of ZGDV/Fraunhofer IGD and Holzmann AG

X Aid for assembly in car manufacturing: instructions
for two-handed screw fixing
Image: courtesy of ZGDV/Fraunhofer IGD, data courtesy of BMW

X Aid for assembly in car manufacturing: lever inside the
door to be pushed to right position
Image: courtesy of ZGDV/Fraunhofer IGD, data courtesy of BMW

August 16, 1998 ICPR 98 10
More Possible Applications
X Computer Aided Surgery
X Fashion Design or Apparel Retail
X Circuit Board Diagnostics and Repair
X Facilities Maintenance
X Industrial Plant Maintenance
X Road Repair and Maintenance
August 16, 1998 ICPR 98 11

Major Components
X Display subsystem
X Rendering subsystem
X Video input and image understanding
subsystem
X Interaction subsystem
X Registration and tracking subsystem
August 16, 1998 ICPR 98 12

Display Technologies
See-through video display technology
Scan conversion
video
and video keying
Graphics
Workstation graphics Mixed Video
and Graphics
August 16, 1998 ICPR 98 13

Display Technologies
X See-through Head Mounted Displays (see-through
HMD)
X World seen directly by the eye with graphics
superimposed
Graphics
Workstation
August 16, 1998 ICPR 98 14
Rendering subsystem
X VR technology
– 3D real-time rendering
– 3D interaction
– head and camera tracking
August 16, 1998 ICPR 98 15

Virtual Camera
X 3D graphics rendered through a virtual
camera
X Realistic augmentation (with registration
and correct optics)
==> virtual camera matches real camera or optics
of human eye.
August 16, 1998 ICPR 98 16

Coordinate Systems in see-
through video
A
World
Camera Coordinates
Coordinates
Pointer
Coordinates
D E
G C
1 Camera
Mark G Pointer G
3 2
F Mark
1
F Object
3
Mark
F
Tracker 2 Object
Transmitter Coordinates
Coordinates
August 16, 1998 ICPR 98 17
Calibration
X All the coordinate transforms in the
previous slide need to be estimated.
X This is done through calibration.
August 16, 1998 ICPR 98 18

Image Calibration
Screen
X Scan converter can
access an arbitrary
region of the screen
to be mixed with
graphics.
X Modeled as 2D
translation and
scaling
August 16, 1998 ICPR 98 19

Image Calibration
Image of two known points displayed
and the result is captured
x c
A A
y r
B B
Computer Generated image Grabbed image
Calibration estimates kx, ky, tx, ty by equations
~
c = kx x + tx k x = ( c B − c A ) /( x B − x A )
~
k y = ( rB − r A ) /( y B − y A )
r = ky y + ty ~
tx = cA − k x xA
~
t y = rA − k y y A
August 16, 1998 ICPR 98 20
Camera Calibration
X Estimates the initial
pose of the camera:
R: rotation matrix
T: translation Camera being
X Estimates the camera calibrated
intrinsic parameters:
f: focal length Calibration jig
( r0 , c0 ): image center
( sx , s y ): scale factors
August 16, 1998 ICPR 98 21

Camera Calibration (continued)
X Many camera calibration methods exist in
computer vision literature:
– R. Tsai
– Weng, Cohen, Herniou
–…
X The estimated camera parameters are used
in the virtual camera with which 3D
graphics are rendered.
August 16, 1998 ICPR 98 22
Points are picked from
calibration jig whose 3D
coordinates are known
The model of calibration

jig is rendered using a
virtual camera with the
estimated parameters.
The rendered graphics is
superimposed on the image
of calibration jig.
August 16, 1998 ICPR 98 23

( r0 , c0 ) : image center
( su , sv ) : scale factors in x and y (or u and v) directions
f : focal length
[ ]
R = rij : rotation matrix for the camera
 t1 
T = t 2  : translation vector for camera
t3 
August 16, 1998 ICPR 98 24

Camera Equations given by:
r − r0 r − r0 r11 x + r12 y + r13 z + t1

= =
su f fu r31 x + r32 y + r33 z + t3
c − c0 c − c0 r21 x + r22 y + r23 z + t 2
= =
sv f fv r31 x + r32 y + r33 z + t3
(r , c) : image coordinates in rows and columns
August 16, 1998 ICPR 98 25

X Collect image data points (ri , ci) corresponding
to known 3D calibration points (xi ,yi ,zi )
X These coordinates are corrected using the
result of image calibration.
X Substitute into camera equations in previous
slide ==> gives 2 equations per calibration
point.
August 16, 1998 ICPR 98 26

X Variable substitution to linearize camera
equations:  R T1 
  T
 R = R 2 
W1 = f u R1 + r0 R 3 
  T
  R
 3
W2 = f v R 2 + c0 R 3  where 
   wi1 
W3 = R 3  W =  w 
 i  i2 
w4 = f u t1 + r0t3   wi 3 
w5 = f vt 2 + c0t3
w6 = t3
August 16, 1998 ICPR 98 27
Solve the homogeneous equation
AW = 0
where
 − x1 − y1 − z1 0 0 0 r1 x1 r1 y1 r1 z1 −1 0 r1 
 0 0 0 − x1 − y1 − z1 c1 x1 c1 y1 c1 z1 0 − 1 c1 

A= 
 
 − xn − yn − zn 0 0 0 rn xn rn yn rn z n − 1 0 rn 
 0 0 0 − xn − yn − zn cn xn cn y n cn z n 0 − 1 cn 
August 16, 1998 ICPR 98 28

Pointer calibration
X 3D pointer is implemented Pointer World
Coordinates Coordinates
with magnetic tracker.
X The tracker coordinate
system and the pointer tip Pointer
needs to be related to the Mark
world coordinate system. coordinates
X This is done through a

pointer calibration process.
Tracker
Transmitter
Coordinates
August 16, 1998 ICPR 98 29

Tracker and pointer calibration
X Calibration done by
picking points from
the calibration jig.
Known points picked
Some points are picked

with different orientations
August 16, 1998 ICPR 98 30
Pointer Calibration
The tip of the pointer is Receiver coordinate
given by the equation: pT system orientation: RM
pW = pM + RM pT pM
Reading the pointer at n different
orientations, we get the equation
pW
I − RM1   pM 1 
   
I − RM 2  pW   pM 2 
   =  


  pT 



I 
− RM n   
  pM n  Tracker
Solve the overdetermined system for Transmitter
pT and pW Coordinates
August 16, 1998 ICPR 98 31

Tracker transmitter calibration
X Estimate position of
transmitter in world P
coordinates
J
X Read three known points on L
calibration jig.
X The origin of the world, p0,
with respect to the
transmitter is solved from pwJ = pM J + RM J p0
after the rotation RMJ is
estimated
August 16, 1998 ICPR 98 32
Tracker Transmitter Calibration
X The rotation matrix RMJ is
estimated by first P
computing the axes: J
x = pWL − pWJ L
z = pWP − pWJ
y=
z
×
x x z T
z x
RM J =[ y ]
x z
August 16, 1998 ICPR 98 33

Registration
X Aligning 3D models of objects with their
real counterparts.
X Alignment errors must be extremely small.
August 16, 1998 ICPR 98 34

Registration (examples)
Example applications where registration is used
A 3D model of the table …obtain the effect of occlusion

is registered with the by using the geometry of the 3D
physical table to... model
August 16, 1998 ICPR 98 35
Registration (Examples)
A 3D model of the engine is … label the parts of the engine

registered with the physical using the parts knowledge
engine to... encoded in the 3D model.
August 16, 1998 ICPR 98 36

Registration Methods
X Image based landmarks for computing object
pose.
X 3D pointer based for computing object pose.
X Automatic registration of objects in 2D
images using computer vision techniques.
August 16, 1998 ICPR 98 37

Image Based Landmarks
X Camera parameters known from camera
calibration.
X 3D coordinates of landmarks known from
object model.
X 2D image coordinates of landmark points
picked manually.
August 16, 1998 ICPR 98 38

X Plug the known values into the camera
equations:
(ri − r0 ) xi r31 + (ri − r0 ) yi r32 + (ri − r0 ) zi r33
+ (ri − r0 )t3 − f u xi r11 − f u yi r12 − f u zi r13 − f u t1 = 0
(ci − c0 ) xi r31 + (ci − c0 ) yi r32 + (ci − c0 ) zi r33
+ (ci − c0 )t3 − f v xi r21 − f v yi r22 − f v zi r23 − f v t 2 = 0
X Solve for translation and rotation under the

constraint of orthonormal rotation matrix.
August 16, 1998 ICPR 98 39
X Solution obtained by minimizing
2
Ax + α R R − I
2 T
where x is the vector of unknowns.
August 16, 1998 ICPR 98 40

3D Pointer Based Landmarks
X Calibrated 3D pointer device,
X 3D coordinates of landmark points known
from the object model,
X 3D landmark points picked on the physical
object with the pointer.
August 16, 1998 ICPR 98 41

X The picked 3D landmark points and 3D
model landmarks are related to each other
with a rigid transformation:
p = Rp + T
W
i i
L
for i = 1, , n
where
piW : world coordinates
piL : landmark coordinates
August 16, 1998 ICPR 98 42

X Solution by minimizing equation:
2 2
p − Rp − T + α R R − T
W
i i
L T
August 16, 1998 ICPR 98 43

Automatic Registration
X Computer Vision techniques for registering
objects with their images and computing
object pose
X Features detected automatically and
matched to model
X Less general
X Less robust
August 16, 1998 ICPR 98 44

Registration Pitfalls
X R matrix in the solution to these equations
may not be a valid rotation matrix.
X One possible approach is to formulate the
equations representing the rotation as a
quaternion instead of a 3x3 matrix.
August 16, 1998 ICPR 98 45

Tracking subsystem
X Many tracking system possibilities
– Magnetic trackers
– Mechanical trackers
– Optical trackers
– Ultrasound trackers
– Vision based trackers
August 16, 1998 ICPR 98 46

Magnetic trackers
X Objects are tracked by attaching a receiver of
magnetic trackers onto the objects.
X The data read from the magnetic tracker is used to
update the object-to-world rigid transformation.
X This requires a calibration procedure (as part of
object registration) by which the object-mark-to-
object transformation is initially estimated.
August 16, 1998 ICPR 98 47

Vision Based Tracking
X Camera tracking
X Object tracking
– One implementation using landmarks put in the
environment. Camera motion is computed
automatically.
August 16, 1998 ICPR 98 48

Vision Based Camera Tracking
X Environment modified
with landmarks
whose 3D coordinates are
known
X Landmarks detected
automatically
X camera pose extracted
from corner coordinates of
square landmarks similar
to camera calibration
procedure.
August 16, 1998 ICPR 98 49

X Landmark squares in the environment are
detected (segmented) using a watershed
transformation algorithm
Original image Watershed Results of inside

transformation operation for regions
of watershed transformation
August 16, 1998 ICPR 98 50

X Landmark squares contain red squares at
known positions which encode the identity
of the model squares
Red squares are barely visible in the
green and blue channels.
Black squares segmented in the green

channel.
Red channel is sampled at locations
where red dots should be w.r.t. the
landmark boundary giving the id of the
square.
August 16, 1998 ICPR 98 51
Squares detected and
room scene augmented
with a wall rendered
with proper perspective
August 16, 1998 ICPR 98 52

X Motion model assumed:
p = vc + ω × ( p − c)
where c is the center of mass of the object,
p is a point on the object,vc is the
translational velocity at c, and ω is the
angular velocity around an axis through c.
August 16, 1998 ICPR 98 53

X Extended Kalman filter employed for
optimal pose and motion estimation.
August 16, 1998 ICPR 98 54

Results
August 16, 1998 ICPR 98 55

X Example video clip
showing camera
tracking with
augmentation of the
scene
August 16, 1998 ICPR 98 56

X Occlusions of
landmarks tolerated as
long as at least two
landmarks are
completely visible.
August 16, 1998 ICPR 98 57

Vision Based Tracking
X Objects can also be tracked using
landmarks on them that can be detected
automatically.
X Other vision-based tracking methods may
also be used such as optical flow.
August 16, 1998 ICPR 98 58

Hybrid Tracking
X Hybrid methods for tracking have been
developed for augmented reality
– UNC research: combination of magnetic
trackers and vision based landmark trackers
– Foxlin: combining inertial trackers with
ultrasound.
August 16, 1998 ICPR 98 59

Interaction of Real and Virtual
Objects
X Occlusions
X Interactive queries (e.g., 3D pointer)
X Collisions
X Shadows and lighting effects
August 16, 1998 ICPR 98 60

Interactions of Real and Virtual
Objects: Occlusion
X Method 1: model-based
– registering a model with the physical object
– rendering the object in black so graphics can be
chroma-keyed
– use z-buffer to occlude
August 16, 1998 ICPR 98 61

Objects: Occlusion
X Method 2: geometry extraction
– compute dense depth map
– use the computed depth map in z-buffer to
occlude virtual objects.
August 16, 1998 ICPR 98 62

Objects: Occlusion
X Dense depth map can be computed using
stereopsis (or other range sensors)
Left Image Right Image

August 16, 1998 ICPR 98 63
Objects: Occlusion
Computed Dense Depth Depth Map used as z-buffer

Map (darker w closer) to obtain occlusion effect
August 16, 1998 ICPR 98 64

Objects: Collision
X A synthetic object placed in a real scene
must give the perception of physical
interaction with its environment w
collision detection is an important problem.
August 16, 1998 ICPR 98 65

Objects: Collision
X Requirements:
– geometry of the synthetic objects is known
– geometry of the scene is known
X Classical methods of collision detection can
be utilized based on geometry intersection.
Ref: John Canny
wTime consuming
August 16, 1998 ICPR 98 66

Objects: Collision
X Method 2: use z-buffer techniques
– Some applications are sufficiently constrained
that they can cheat and still have a realistic
effect.
– Scene geometry can be used to build a z-buffer,
which then can be used to check for simple
collision tests with the synthetic objects.
– Ref: Breen et al.
August 16, 1998 ICPR 98 67

Illumination
X Realistic mixing of synthetic objects with
images of the scene also involve:
– Rendering the synthetic objects with the same
illumination as the actual scene w extract
illumination sources in the scene
– Having objects cast shadows on each other in a
correct manner w extract illumination as well
as geometry of the scene.
August 16, 1998 ICPR 98 68

Illumination
X The effect of illumination is summarized by
the equation:
L pixel (c ) = k a (c) Lia (c)π
+ k d (c) ∫ L(c)( N ⋅ L)dω
ω
+ k s (c) ∫ L(c)( N ⋅ H ) dω
n
where c ∈ {R, G , B} is the color channel

August 16, 1998 ICPR 98 69
Illumination
X Most illumination estimation methods
utilize some form of shape-from-shading
formulation from the diffuse component of
the previous equation.
August 16, 1998 ICPR 98 70

Illumination
Normal N
Light Source L
I = ρλ ( N ⋅ L)
Viewer V
X ρ : albedo of the surface

X λ : flux per unit area perpendicular to
incident rays
X Ref: Pentland PAMI 1984
August 16, 1998 ICPR 98 71

Illumination
X In a local homogeneous area of the surface,
albedo and illumination change little
X This assumption gives us:
dI = d ( ρλ ( N ⋅ L))
= ρλ (d N ⋅ L) + ρλ ( N ⋅ d L)
= ρλ (d N ⋅ L)
August 16, 1998 ICPR 98 72

Illumination
X Assumption: changes in surface orientation are
isotropically distributed.
X Look at the effect of illuminant direction on
dI , the mean value of dI.
dI = ρλ (d N ⋅ L )
= ρλ ( x L d x N + y L d y N + z L d z N )
(d x N , d y N , d z N ) : mean change in d N measured along direction ( dx, dy )
( xL , y L , z L ) : light source direction
August 16, 1998 ICPR 98 73

Illumination
X Assuming change in surface normal is
isotropic, then
d x N is proportional to dx
d y N is proportional to dy
d zN = 0
and
d I = k ( xL dx + y L dy )
August 16, 1998 ICPR 98 74

Illumination
We can estimate illumination direction
from the last equation
n by measuring the mean of dI along a
number of directions
o and setting up a linear regression
formulation to estimate light source
direction
August 16, 1998 ICPR 98 75

Illumination
X Integrated methods for estimating
illumination direction with shape-from-
shading also exist
– Ref: Hougen + Ahuja, ICCV 93, Berlin
X More recent works of illumination
extraction:
– Ref: Walter, Alppay, et al, Siggraph ‘97
August 16, 1998 ICPR 98 76

Illumination and Interaction of Real
and Virtual Objects: Shadows
X Shadows cast by real objects on virtual
objects and vice versa increase the realism
in some applications
– scene and object geometry should be known
– illumination model should be known
– then by rerendering the scene one can compute
the shadows in the mixed reality scene
– Ref: Fournier
August 16, 1998 ICPR 98 77

Calibration-free Augmented
Reality
X Display and overlay of graphics is done
without extraction of camera parameters.
X Basic operation is reprojection:
– Given: Four or more 3D points,
» the projection of all the points in the set can be
computed as a linear combination of the projection
of just four of the points.
X Ref: Kutulakos & Vallino in TVCG Jan-March 1998.
August 16, 1998 ICPR 98 78

Reality
ψ χ Affine bases points + p0
b1 b2
are the fiducial points
p0 b3 tracked that define the affine
representation
ζ : Affine viewing direction
Affine coordinates
X Reprojection property of p on an object
 u  ub1 − u p0 ub2 − u p0 ub3 − u p0 u p0   x 
 v  v − v vb2 − v p0 vb3 − v p0 v p0   y 
  =  b1 p0
 w  ςT z p0   z 
    
1  0 0 0 1  1 
August 16, 1998 ICPR 98 79

Reality
X Affine coordinates of a 3D point p can be
computed from their projections along two
different viewing directions.
u1p  ub11 − u1p0 ub12 − u1p0 ub13 − u1p0 u1p0   x 
 1  1 
 v p  =  vb1 − v1p0 vb12 − v1p0 vb13 − v1p0 v1p0   y 
 
u 2p  ub2 − u 2p0 ub22 − u 2p0 ub23 − u 2p0 2 
u p0  z 
 2   21 2  
 v p   vb1 − v 2p0 vb22 − v 2p0 vb23 − v 2p0 v p0   1 
Affine coordinates
Solve for x, y, and z Known from coodinates in
two images
August 16, 1998 ICPR 98 80
Reality
X The affine representation of a 3D object can
be used by tracking 4 fiducial points across
frames and using their image coordinates to
compute reprojection of the other points on
the objects.
August 16, 1998 ICPR 98 81

Future Research Topics
X Calibration of see-through HMD’s
– Optics are different (human eyes are in the
loop)
– There is no explicit image from which points
are to be picked directly.
– Calibration must be done using some other,
possibly interactive technique.
August 16, 1998 ICPR 98 82

Future Research Topics
X Increased accuracy and automation in
– object registration, and
– tracking (camera and/or object)
X New display technologies
– for example, projection of graphics onto real
scenes
August 16, 1998 ICPR 98 83

Resources
X My Web page address:
http://www.cs.iupui.edu/~tuceryan/AR/AR.html
X Augmented Reality Page by Vallino
http://www.cs.rochester.edu/u/vallino/research/AR
Many links to other places from here.
X Fraunhofer AR page:
http://www.igd.fhg.de/www/igd-a4/ar/
X MIT AI Lab image guided surgery page:
http://www.ai.mit.edu/projects/vision-
surgery/surgery_home_page.html
August 16, 1998 ICPR 98 84

Augmented Reality: Integrating Computer Graphics With Computer Vision Mihran Tuceryan

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Augmented Reality: Integrating Computer Graphics With Computer Vision Mihran Tuceryan

Uploaded by

Copyright:

Available Formats

Augmented Reality

Integrating Computer Graphics with

August 16, 1998 ICPR 98 1

August 16, 1998 ICPR 98 2

Electric wires shown

August 16, 1998 ICPR 98 3

August 16, 1998 ICPR 98 4

Image: courtesy of ZGDV/Fraunhofer IGD and Holzmann AG

August 16, 1998 ICPR 98 8

Image: courtesy of ZGDV/Fraunhofer IGD, data courtesy of BMW

Image: courtesy of ZGDV/Fraunhofer IGD, data courtesy of BMW

August 16, 1998 ICPR 98 11

August 16, 1998 ICPR 98 12

August 16, 1998 ICPR 98 13

August 16, 1998 ICPR 98 15

August 16, 1998 ICPR 98 16

August 16, 1998 ICPR 98 18

August 16, 1998 ICPR 98 19

August 16, 1998 ICPR 98 21

The model of calibration

August 16, 1998 ICPR 98 23

August 16, 1998 ICPR 98 24

r − r0 r − r0 r11 x + r12 y + r13 z + t1

August 16, 1998 ICPR 98 25

August 16, 1998 ICPR 98 26

August 16, 1998 ICPR 98 28

X This is done through a

August 16, 1998 ICPR 98 29

Known points picked

Some points are picked

August 16, 1998 ICPR 98 31

computing the axes: J

August 16, 1998 ICPR 98 33

August 16, 1998 ICPR 98 34

A 3D model of the table …obtain the effect of occlusion

A 3D model of the engine is … label the parts of the engine

August 16, 1998 ICPR 98 36

August 16, 1998 ICPR 98 37

August 16, 1998 ICPR 98 38

X Solve for translation and rotation under the

where x is the vector of unknowns.

August 16, 1998 ICPR 98 40

August 16, 1998 ICPR 98 41

August 16, 1998 ICPR 98 42

August 16, 1998 ICPR 98 43

August 16, 1998 ICPR 98 44

August 16, 1998 ICPR 98 45

August 16, 1998 ICPR 98 46

August 16, 1998 ICPR 98 47

August 16, 1998 ICPR 98 48

August 16, 1998 ICPR 98 49

Original image Watershed Results of inside

August 16, 1998 ICPR 98 50

Black squares segmented in the green

August 16, 1998 ICPR 98 52

August 16, 1998 ICPR 98 53

August 16, 1998 ICPR 98 54

August 16, 1998 ICPR 98 55

August 16, 1998 ICPR 98 56

August 16, 1998 ICPR 98 57

August 16, 1998 ICPR 98 58

August 16, 1998 ICPR 98 59