Markerless Monocular Motion Capture Using Image Features and Physical
Constraints
Yisheng Chen∗ Jinho Lee†
The Ohio State University MERL
A BSTRACT
We present a technique to extract motion parameters of a human
figure from a single video stream. Our goal is to prototype motion
synthesis rapidly for game design and animation applications. For
example, our approach is especially useful in situations where mo-
tion capture systems are restricted in their usefulness given the var-
ious required instrumentation. Similarly, our approach can be used
to synthesize motion from archival footage. By extracting the sil-
houette of the foreground figure and using a model-based approach,
the problem is re-formulated as a local, optimized search of the pose
space. The pose space consists of 6 rigid body transformation para-
meters plus the internal joint angles of the figure. The silhouette of
the figure from the captured video is compared against the silhou-
ette of a synthetic figure using a pixel-by-pixel, distance-based cost
function to evaluate goodness-of-fit. For for a single video stream,
this is not without problems. Occlusion and ambiguities arising
from the use of a single view often cause spurious reconstruction of
the captured motion. By using temporal coherence, physical con-
straints, and knowledge of the anatomy, a viable pose sequence can
be reconstructed for many live-action sequences.
CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional
Graphics and Realism-Animation— [I.4.9]: Image Processing and
Computer Vision—Applications
Keywords: computer animation, model-based reconstruction
1 I NTRODUCTION
Motion Capture (mocap) has become a mainstay of many computer
animated productions and is often used to capture human motion.
Commercial mocap systems capture motion by fixing on the target
figure instrumentation such as optical, magnetic, or mechanical sen-
sors, passive optical reflectors or active optical emitters. However,
various constraints imposed by the instrumentation tend to limit the
usefulness of mocap either by restricting the physical space of the
movement, restricting the environment in which motion can be cap-
tured, or restricting the movement itself (not to mention restrictions
due to cost).
While motion libraries and motion retargetting techniques extend
the usefulness of mocap, there is a need to develop algorithms and
methods for commodity sensors. For example, there is often a need
to synthesize the motion of a figure in a video clip obtained from the
Web, or from a surveillance camera. Conversely, one would like to
capture human motion with the least amount of equipment possible
- using a single consumer-grade camera. Additionally, we wish to
produce results at rates that would make such a system useful in
an interactive environment and therefore suitable for prototyping
animated sequences and exploring initial character motions.
∗ e-mail:chenyis@cse.ohio-state.edu
† e-mail:leejh@merl.com
‡ e-mail:parent@cse.ohio-state.edu
§ e-mail:raghu@cse.ohio-state.edu
Rick Parent‡ Raghu Machiraju§
The Ohio State University The Ohio State University
However, creating a single-camera, interactive, markerless system
to capture human motion is challenging [6]. On the other hand,
there has been a concerted effort by several researchers to develop
methods that will reconstruct motion recorded by either a limited
number of cameras or just a single camera.
Generally speaking, capturing human motion from video involves
extracting features from video sequences and matching those fea-
tures to some model or representation of motion to be reconstructed.
Data-driven approaches exploit a motion database for reconstruct-
ing target motion (e.g., [11]). Such strategies can provide real-time
albeit limited reconstruction of motion. Motion outside the confines
of the database is constructed with lesser fidelity. Other approaches
key on a specific type of motion, such as cyclic, sagittally symmet-
ric motion, and look for specific indicators of motion phases (e.g.,
[2]). In a more general vein, motion-templates are used to recognize
a variety of motions (e.g., [1]). More general are systems that use
a 3D model of the human figure to recreate a figure’s pose for each
frame. Rehg and his associates [5], and Sminchisescu [19, 21, 20]
have used both 3D models of humans to emulate and synthesize
motion as depicted in a single video stream. Our approach also
falls into this last category. We now elaborate on our model-based
approach.
1.1 3D Model-based Approach
Model-based approaches use a 3D articulated model of the human
body to estimate the pose and shape such that the model’s 2D pro-
jection closely fits the captured person in the image. Features such
as intensities, edges, silhouettes are widely used. Processing color
and texture information is computationally expensive and changes
in illumination present significant problems. It should be noted
much of the scene and camera information is unknown. As a re-
sult, many of the efforts to track human figures in video, including
ours, use silhouettes. Silhouettes are less sensitive to noise than
edges but fine details might be lost in its extraction.
The use of silhouettes does create problems. Extracting pose from
a silhouette can produce multiple candidate solutions. The high
dimensionality of the articulated model parameter space requires
efficient yet robust search algorithms. A suitable initialization of
model parameters is also required by many algorithms for motion
tracking. Constraints like joint angle limits on parameters can be
used as well as knowledge about physics and the anatomy.
By extracting the silhouette of the foreground figure and using a
model-based approach, the problem is re-formulated as a local, op-
timized search of the pose space. The pose space, in turn, consists
of 6 rigid body transformation parameters plus the internal joint
angles of the figure.
Occlusion and ambiguities arising from the use of a single view of-
ten cause spurious reconstruction of the captured motion. There-
fore, the reconstruction is successful when either 2D correspon-
dence is established for each frame of the sequence manually [5],
or when the motion under scrutiny is limited [19, 21]. There is a
paucity of methods that allow for general motion and require little
manual intervention. Additionally, these methods should be effi-
cient.
We describe herein a method to reconstruct arbitrary motion se-
quences that is model-based, operates on images, and exploits
knowledge about the anatomy. Consequently, our method is simple,
efficient and requires limited manual intervention. Will our method
reconstruct all motion sequences successfully? The answer is no.
Occlusion of limbs by larger parts of the anatomy cannot always be
resolved through the use of image silhouettes. On the other hand,
we wish to explore the limits of efficient monocular motion recon-
struction. Our results show that we can reconstruct increasingly
complex motion when we include a larger set of anatomical fea-
tures and constraints and employ robust image comparison metrics.
We now provide an overview of our approach.
1.2 Overview of Our Approach
In this paper, we explore inverse methods which reconstruct motion
from a single camera. We examine techniques which use silhouettes
and edges rather than texture. Since motion is our primary interest
and not identification, silhouettes and edges provide ample grounds
for developing robust methods. Inverse design methods like ours
employ optimization algorithms to minimize an objective function.
Resolving occlusion between various parts of the human body is
certainly an ominous challenge given the difficulty of matching an
imperfect, highly flexible, self-occluding model to cluttered image
features. Viable human models have at least 20 joint parameters
subject to highly nonlinear physical constraints. Also, a significant
number of the possible degrees of freedom afforded by the vari-
ous joints are not uniquely determined by any given image. Thus,
monocular motion capture is ill-posed and non-linear. Methods re-
ported in the literature are either complex and expensive for com-
puter graphics applications or impose severe restrictions on the type
of motion that can be captured.
The work described herein is an exploration of simple yet robust
cost functions and incremental search strategies. Our focus on sim-
pler cost functions will eventually allow for the realization of near
real-time reconstruction often needed for computer graphics appli-
cations while at the same time making few assumptions about the
motion being tracked.
The starting point of our method is a model of a human actor with
multiple quadrics assembled at various joints. After an initial pose
is established either automatically or with the aid of the user, a
frame-to-frame tracking procedure ensues in which the solution of
the last frame is the initial guess for the next frame. In addition to
silhouettes and edges, our objective function uses anatomical and
physical constraints to aid in disambiguating the view.
The main contributions of our work include:
• development of a new core-weighted XOR metric for model
localization in an image
• robust detection and tracking of body parts such as head,
arms, feet and the V between the legs in the image
• inclusion of image features and anatomical constraints to re-
duce the size of optimal search space
Our new method is shown to be capable of reconstructing full ar-
ticulated body motion efficiently. Additionally, our results include
video sequences of varying length and arbitrary illumination and
the reconstruction is quite faithful to the original sequence.
The paper is organized as follows. In Section 2 we describe perti-
nent previous work in motion reconstruction. Our human model is
briefly discussed in Section 3. We describe in detail our method for
motion reconstruction in Section 4. Implementation considerations
are given in Section 5. Section 6 includes results which demon-
strate the effectiveness of our technique. Concluding remarks and
pointers to future work are described in Section 7.
2 P REVIOUS W ORK
We describe work that is closely related to our own . Model based
approaches use a 3D articulated model of the human body to esti-
mate the pose and shape such that the model’s 2D projection closely
fits the captured person in the image. Bregler and Malik [2] propose
a framework to estimate 3D poses of each body segment in a kine-
matic chain using twist representation of a general rigid-body mo-
tion. However, their reconstruction assumes lateral symmetry in the
tracked motion. Pavlovic et al [16] present a system to track fronto-
parallel motion using a dynamic Bayesian network approach. Their
work focuses on the dynamics of human behavior as described by
bio-mechanical models.
Deutscher et al [4] use a human model to build a framework of a
kinematic chain using limbs of conical sections for computational
simplicity and high-level interpretation of output. They use edges
and silhouettes for their cost function to estimate pose from multi-
ple camera views. A condensation algorithm is employed to search
the high dimensional space without restrictions. Carranza et al [3]
use silhouettes from multi-view synchronized video footage to re-
construct the motion of a 3D human body model, and then re-render
the model’s appearance and motion interactively from any novel
view point. Kakadiaris et al [10] use a spatio-temporal analysis to
track upper body motions from multiple cameras.
Monocular markerless motion capture has been studied by a few re-
searchers. Sminchisescu and Triggs [20] achieve successful motion
synthesis based on the propagation of a mixture of Gaussian den-
sity functions, each representing probable 3D configurations of the
human body over time. Difranco et al [5] propose a batch frame-
work to reconstruct poses from a single view point based on 2D
correspondences subject to a number of constraints. Their methods
are shown to be successful when deployed on moderately difficult
sequences that include athletic and dance movements.
Alternative approaches to model based tracking of human bodies
have been also used. Bobick and Davis [1] construct temporal tem-
plates from a sequence of silhouette images and present a method
to match the temporal template against stored views of the known
actions. Wren et al [22] use 2D blobs for tracking motion in a video
image. Leventon and Freeman [14] take a statistical approach and
used a set of motion examples to build a probability model for the
purpose of recovering 3D joint angles for a new input sequence.
Lee et al [11] present a vision-based interface to control avatars in
virtual environments. They extract visual features from the input
silhouette images and search for the best motion matching the vi-
sual features obtained from rendered versions of actions in a motion
database. Ren et al [18] use the AdaBoost algorithm to select a few
best local features from silhouette images to estimate yaw and body
configurations. Finally, a survey of computer vision-based human
motion capture techniques is presented by Moeslund and Granum
[15].
We reconstruct general 3D motion from the silhouettes extracted
from a single view without relying on a motion database. As a
consequence, our work is closest to that of Sminchisescu. However,
we demonstrate the tracking of motions that are more complex than
those presented by Sminchisescu and we track at speeds that are
equal or faster than those reported.
3 O UR H UMAN M ODEL
Our human model is a combination of spheres and cylinders with
anisotropic scale and rigid transformation for an object coordinate
system for each part. The model is shown in Figure 8 and Figure 9.
The parameters that describe our human model consist of two com-
ponents - shape and motion parameters. We define twelve shape
parameters which describe the scale factors to be multiplied to a
predefined ‘standard’ size of each body part. Motion parameters
are composed of 6 global transformation and total 24 joint angles.
Each joint has anywhere from one to three degrees-of-freedom.
Given N frames from a video sequence, our human body model
at a specific frame i is represented by M (i) = M(α , β (i) ), where
α = {α1 , α2 , . . . , αm } (m = 12) is the shape parameter vector and
(i) (i) (i)
β (i) = {β1 , β2 , . . . , βn } (n = 30) is the motion parameter vector.
4 T HE O PTIMIZATION S TRATEGY
The use of a single view makes the reconstruction problem ill-
posed, as stated earlier. We now describe our optimization strat-
egy to fit the motion parameters to a single stream of silhouette
images. First, we describe how we extract silhouettes from a video
sequence. Then, we present an objective function based on the dif-
ference of area between model-generated silhouette and input sil-
houette. Next, we discuss how we improve the performance of our
optimization algorithm. We achieve this by incorporating edge in-
formation along with both physical and anatomical constraints into
the objective function. Finally, we discuss a non-linear multidimen-
sional optimization algorithm we employ to minimize the proposed
objective function.
4.1 Silhouette Extraction
The input to our motion synthesis system is a sequence of silhouette
images that describe the gross motion of the human body. We avoid
using coloration or texture information in order to minimize the ef-
fects of variable viewing conditions. In addition to using silhou-
ettes we also employ high-contrast edges to further our optimiza-
tion strategy as explained below. Given a video footage, there exist
several methods to obtain silhouette images. Although silhouettes
are less sensitive to noise than edges, fine details of body structure
and motion are often lost in their extraction. We employed the fol-
lowing semi-automatic methods to extract the foreground human
figures from the background.
First, we identify frames wherein the human figure is absent and
construct a statistical model of the background. The mean and stan-
dard deviation at all pixels over all frames (and for every color RGB
channel) is computed. If a pixel differs too much from the back-
ground on any color channel, the pixel is treated as a foreground
pixel and thus the silhouette is recovered in a pixel-wise manner. In
case, there exists no image with just the background, we identify
frames that describe motion of figures moving through the scene
in a predictable fashion. We then compute a median image over
the whole sequence followed by building a weighted-mean/variance
background model to extract the silhouette out, if over time there
will more non-motion at a pixel than there will be motion [9]. Oth-
erwise, we select several frames with little overlap of the human
figures, extract the figures manually, and then combine the result-
ing images to obtain a composite background image. If there exists
background areas that cannot be recovered, they are considered as
foreground anyway.
Once we derive a background model, we can subtract the back-
ground from all the images in the sequence. Often, we do not have
a perfect background model, and therefore the silhouette images
may suffer from the presence of noise. The silhouette quality can
be improved using morphological operations, such as dilation and
size-based object filtering.
4.2 Core-Weighted XOR
Our goal is to find the motion parameter set β that minimizes the
total penalty
(i)
E(β (i) ) = f (Sinput , Smodel (β (i) )) (1)
(i)
for a suitable cost function f , where Sinput and Smodel (β (i) ) are the
ith input silhouette image and a silhouette image generated by M (i)
(i)
respectively. For the sake of clarity, we instead denote Sinput and
Smodel (β (i) ) simply as Sinput and Smodel respectively.
How does one design a viable and robust cost function f as de-
scribed in Eq.(1) ? The easiest way to measure the difference of
two binary images is the number of ‘on’ pixels when pixel-wise
XOR operation is applied to the two images [13]. In this case,
H W
f (Sinput , Smodel ) = ∑ ∑ c(i, j) (2)
i j
0 if Sinput (i, j) = Smodel (i, j)

c(i, j) =
1 otherwise.
where the double summation iterates over all pixel locations. If our
goal requires that f = 0, that is, if two silhouettes overlap exactly,
the optimal solution will be unique in terms of Smodel . However, if
our objective function f cannot be reduced to zero given inherent
characteristics of the problem, it is likely that there are multiple
optimal solutions. Any preference among those multiple optimal
solutions should be incorporated into the cost function.
Since limbs are features of particular importance in any articulated
figure, we do not want to lose track of those features. Limbs in a
human body are often well characterized by their skeleton or medial
axis derived from the silhouette image. Therefore, whenever ambi-
guity occurs it would be better to choose the direction in parametric
shape space such that model-generated silhouette covers the core
area of the input silhouette. The core area includes the silhouette
pixels close to the skeleton. This requirement can be incorporated
in the cost function by imposing higher penalty if Smodel (β (i) ) does
not overlap any region near the core area of the input silhouette
(i)
Sinput .
Our new cost function replaces c(i, j) in Eq.(2) with
0 if Sinput (i, j) = Smodel (i, j)

c(i, j) = (3)
d(i, j) otherwise
d(i, j) = D(Sinput )(i, j) + wD(S̃input )(i, j),
where D(S) is the Euclidean distance transform of binary image S
and S̃ is the inverse image of S and w is a weighting factor that
controls the importance of coverage of core area relative to the mis-
match in the region outside of silhouette area.
Note that image d represents a distance map from silhouette contour
and can be computed once in a preprocessing step. Figure 1 depicts
the coremap image d with w = 5.0. We call Eq. 2 along with Eq. 3
core-weighted XOR.

Figure 1: (left) An input silhouette image. (right) The coremap
image used to compute the core-weighted XOR
4.3 Using Image Features and Physical Constraints
The core-weighted XOR objective function is sufficient in many
cases. However, there are also cases in which it falls short or
our performance expectations. Simply adjusting the core-weighted
XOR parameters is insufficient to increase the generality of this
method. In particular, the problems encountered include: ap-
pendages are sometimes not extended to completely fill the silhou-
ette, feet are able to penetrate the ground plane, and joints are al-
lowed to bend backward as well as forward. These sub-optimal
configurations are found because they represent local minima of the
core-weighted XOR function. To remove these spurious local min-
ima and improve our performance, additional terms are included in
the basic objective function.
In order to stretch the limbs to cover silhouette edges, we augment
our objective function with a term that emphasizes matching edges,
Eq.(4).
if Tinput (i, j) = Smodel (i, j) = 1

−we
e(i, j) =
0 otherwise
where T is the image after edges have been extracted from the fore-
ground image using a contrast threshold. This term is summed over
all pixel positions along with the core-weighted XOR function.
In addition to the matching edges term, semantic features are used
for the limbs to fully cover the silhouette. We detect the limbs
(hands and feet) along the contours of the foreground human body
in some frames [12]. Along the contour of the silhouette, we cal-
culate the curvature of each point on the contour. The curvature at
those points is compared against a specified threshold, and points of
high positive curvature are extracted. Using an estimate of the po-
sition of hands from several of previous frames, unlikely hand/foot
positions are further eliminated as determined by the human body
structure.
Moreover, the outlines of the arms and the V shape between two
legs can be detected based on those concave points along edge con-
tours; the neck and the crotch are usually the concave points along
the edge contours, and arms begin from the neck and the V shape
between the legs is formed by the feet and the crotch. However,
since we are only using a single camera, it is hard to distinguish
between the left and right limbs. Therefore, we do not require the
exact matching of the limbs from the model and the silhouettes. The
hand/foot constraint term only focuses on reducing the distance be-
tween the model’s hands/feet and the nearest detected ones.
q consecutive frames during the pre-processing stage, it is because
l = −wl (im − id )2 + ( jm − jd )2 (5)
where (im , jm ) is the limb position on the articulated model, (id , jd )
is the detected foot/hand position, and wl is the weight factor.
Arms and the V shape are treated the same as the edge term,
which means we try to get a maximum matching between detected
arms/legs and those of model’s.
−wal if Rinput (i, j) = Smodel (i, j) = 1

al(i, j) = (6)
0 otherwise
where R is the image with arms/legs detected. This term is summed
over all pixel positions along with the core-weighted XOR func-
tion. We use those two terms to synthesize motions in Figure 8 and
Figure 9.
It should be noted that sometimes there is no proper candidate (both
a hand or a foot), and in those cases we do not require that the
hand/foot constraints be satisfied for particular frames. Moreover,
the position of the head can be detected along the contour using the
horizontal projection of the silhouette [7] and temporal coherence.
The head always lies at a local highest position along the silhou-
ette outline and near the maximum of horizontal projection of the
histogram. Similarly, a head constraint term can be used in the ob-
jective function.
(4)
Figure 2: Three examples for hands/feet, arms, legs and head detec-
tions along silhouette outlines, where head is marked as red, body
tips are white, arms in green and legs in red.
To disallow configurations that include ground plane penetration,
we define a constraint on the limbs of the figure that essentially
evaluates to infinity whenever penetration occurs. We also define
constraints for each joint angle such that C lj ≤ β j ≤ Chj , j = 7..30
and penalize configurations that violate the constraints. These con-
figuration penalties are added to the objective function. These mod-
ifications help guide the silhouette-model matching process to a
physically meaningful parameter set with less ambiguity.
Despite the application of physical constraints, temporal coherence
and knowledge of the anatomy, the 3D reconstruction is still under-
constrained due to self-occlusions and ambiguities.
Instead of exploring complex yet not reliable solutions like texture
detection, we allow users to specify the keyframes interactively
to obtain viable results. Similar solutions to this problem are re-
ported in [5]. Every keyframe has an impact range, and for any
frame inside this range, instead of using the configuration of the
last frame as starting body pose, one uses the interpolation of the
keyframe and the previous frame to start the optimization. Enough
keyframes can always guarantee a good reconstruction, but speci-
fying keyframes is tedious. Our suggestion is the following. When
body features like feet and hands are not discernable for several
occlusions or ambiguities occur. Users should add two keyframes,
one before the occlusions happen and one after the occlusions dis-
appear. Users do not necessarily specify all the 30 parameters; they
specify a subset of the parameters, usually those describing position
of the limbs, or the twisted torso. Figure 3 shows a sequence that
requires keyframing to be reconstructed correctly.
Figure 3: Keyframes are necessarily to reconstruct this sequence con-
taining ambiguities and self-occlusions. Figure 10 shows an example
where this keyframe technique is employed. We specify the left and
right pose as the keyframe poses.
4.4 Biased Downhill Simplex Method
Extracting pose from a silhouette can produce multiple candidate
solutions. The high dimensionality of the articulated model para-
meter space requires efficient local and global search algorithms.
We chose local algorithms given their lower cost. A suitable initial-
ization of model parameters is also required by many algorithms for
motion tracking. Constraints like joint angle limits on parameters
can be used. The movement of the subject can be restricted at the
cost of losing generality. For example, movement symmetric (but
out of phase) to the central sagittal plane is often assumed.
The object function disallows the computation of analytic deriva-
tives in terms of motion parameters. Downhill simplex method [17]
serves very well to minimize the proposed cost function since it re-
quires function evaluation for specific multi-dimensional optimiza-
tion parameters.
The simplex method can be easily adapted to our multi-dimensional
human body model. The initial simplex of n dimensions consists of
n + 1 vertices. Though we use this method for alignment and shape
fitting, we explain it’s workings by only describing reconstruction
(i−1) (i−1)
of the motion. Let the coefficients β (i) = {β1 , · · · , βn } (the
solution of the previous frame) be one of the initial points p0 of the
simplex. We can choose the other remaining n points to be
pi = p 0 + µ i e i , i = 1..n,
where ei ’s are n unit vectors and µi are defined by the character-
istic length scale of each component of β . The movement of this
n-dimensional simplex is confined to be within the motion space
close enough to the configuration of the current frame and there is
no need to perform exhaustive searches beyond certain ranges of
movement between two consecutive frames. To further target the
most relevant parameter space to search, parameter velocity is used
to bias the simplex location. This bias and the size of the simplex
are determined by limits on parametric acceleration that arise from
principles of physics and basic anatomy.
Due to the inherent hierarchical properties of the human model, a
hierarchical optimization algorithm is employed. The complete 30
configuration parameters are divided into 6 sub-groups, as dictated
by anatomical considerations. Starting from the configuration of the
pervious frame, the model’s global translation and rotation trans-
formations are first computed using the downhill simplex method.
Then the torso is appropriately rotated to minimize the error. The
arm and leg positions are computed last, one by one.
To avoid accumulative errors, another simple technique that is often
beneficial is to restart the optimization routine from the solution
found in the current optimization stage. In the following section,
we explore how these strategies are exploited towards finding the
correct motion configuration from silhouette image sequences.
5 I MPLEMENTATION
Our reconstruction method depends on an optimization process ex-
ecuted in a motion parameter space. Toward this purpose we first
perform a shape fitting exercise for the very first frame so that our
model fully explains the silhouette area from a specific view. The
resulting shape parameters are then fixed for the remainder of the
motion tracking process. It should be noted that we use the same
objective function for initial alignment, shape fitting and motion
tracking.
5.1 Automatic Initialization
As mentioned before, the only input we exploit to reconstruct the
3D motion parameters are silhouette images extracted from a video
sequence. We use the area-based metric to compute the likelihood
of our model for a given input silhouette image. Therefore, the
projected shape of a 3D model needs to be as close as possible to
the input silhouette. A suitable frame with no self-occlusions is
used to adapt the shape of our model to the actor being observed.
The optimization procedure is employed for this frame in order to
fix the model’s shape parameters and initial the pose of the model.
Lacking a suitable frame, the shape parameters and initial pose of
the model can be initially set by hand.
The optimization parameters we determine in the initialization
stage is a subset of the parameter set α ∪ β . Let a translation vec-
(1) (1) (1) (1) (1) (1) (1)
tor t = {β1 , β2 , β3 } and let s = {βlarm , βrarm , βlleg , βrleg } be a
subset of motion parameters that define the rotation angles around
z-axis of upper arms and upper legs, respectively We divide the ini-
tialization task into two consecutive optimization steps:
1. Alignment: find t and ŝ such that Eq. 1 is minimized.
2. Shape adjustment: find α and s so that Eq. 1 is minimized
given the optimal {t, ŝ} found at step 1.
The middle and right image in Figure 4 depict the result for the two
steps described above. Note that our core-weighted XOR metric is
helpful to align the shape of the initial model at the center of the
silhouette image. If the initial model was not aligned at the center
of the silhouette, the subsequent shape fitting process would have
failed.
5.2 Motion Tracking
After the initialization described in the previous section, actual mo-
(i)
tion tracking is performed by searching a motion configuration β min
such that it minimizes the objective function, as described in Sec-
(i)
tion 4, for each frame i. The optimal parameter vector, β min found
at frame i is used as the initial guess for the optimization routine of
frame i + 1.
To reconstruct only physically meaningful joint angles, we assign
a constraint for each joint angle as well as the degree of freedom
Figure 4: Automatic initialization step: (left) Initial status. (middle)
After alignment. (right) After shape fitting and ready for motion
fitting.
(DOF) of the joint associated with a body part. A Euler angle is
used to describe the rotation of a joint. For example, an elbow joint
has one DOF with range of (0◦ , 160◦ ). The DOFs are included in
the choice of β ; however, the angle constraints need to be incorpo-
rated during optimization. Considering all terms mentioned in 4.3,
we change the cost function described in Eq. 2 to
 H W
 ∑i ∑ j (c(i, j) + e(i, j) + l + al(i, j))
f (β ) = if Ckl ≤ βk ≤ Ckh for k = 7...30 (7)
∞ otherwise.

where c, e, l and al are given by Eq. 3 Eq. 4, Eq. 5 and Eq. 6. Ckl
and Ckh are the lower and upper bound of a joint angle βk respec-
tively. Assigning constraints for each joint angle is very beneficial
in removing the ambiguity when deciding the next step in searching
downhill direction.
In using the downhill simplex method restarting the minimization
from Curious Lab.
routine always helps to escape the shallow local minima that are
proximal to deeper local minima. Figure 5 shows how effective this
simple repetition is. The example sequence consists of four main
actions of a total of 250 frames. The frame range (0, 100) depicts
the bending of the left arm, range (100, 150) shows the bending of
the right arm, range (150, 200) shows the bending of the left leg,
and finally the frame range (200, 250) depicts the same actor bend-
ing his right leg. The results using no repetition and 2-times rep-
etition show relatively high cost values. Incorrect results are also
obtained. Two such incorrect results from the two tracking exper-
iment at a specific frame (224) are shown in Figure 6. Tracking
with 3-times repetition showed visually correct results for all four
action sequences. In this case, the area with high cost in Figure 6 is
mainly caused by the deformation of the shape of the subject. We
note this by displaying the two frames with corresponding peaks in
the graph. The sources of error are marked illustratively in Figure 7.
The region A, D, and E is from the deformation of the clothes. The
region B is caused by the expanded silhouette due to the motion
blur effect and C is a region with the inherent difference between
model and real silhouette area.
5.3 Motion Refinement
The final tracking result cannot be perfect because: the shape of
the articulated model does not match the actor in the video exactly;
there are occlusions and ambiguities; there are many more DOFs
for each articulated joint. However, our system generates reason-
ably good motion data and allows the users to refine the generated
motion in the post-processing stage. The user can also add new
constraints to reduce the search space for particular parameters to
Figure 5: Effectiveness of simple repetition of the same optimization
process.
Figure 6: First: input image; Second: a correct result (with
cost value 1231); Third and forth image shows the 2 incorrect
results (with cost value 2167 and 2130 respectively)
get better tracking results. In addition, given our motion data, user
can modify the motion with any 3D character animation tool, like
R
Poser
6 R ESULTS
We illustrate our method using several full body tracking se-
quences. All the video clips are sampled at the rate of 30Hz, and
some of them are from CMU Graphics Lab Motion Capture Data-
base [8]. Despite that we are using only one camera and do not em-
ploy any marker, we still obtain desirable results. Figure 8, Figure 9
and Figure 10 are examples of tracking artists and athletes. The first
dance sequence (Figure 8) is relatively straightforward to track be-
cause there are no occlusions and motion is mainly planar. Our
tracker manages to track the fronto parallel motions. The second
badminton sequence (Figure 9) is more complex because of multi-
ple occlusions occurring during the action. However we can recon-
struct the motion when we assume that both legs move with con-
stant velocity. In addition, we capture the in-depth (out-of-plane)
motions of the player successfully. The previous sequences could
all be tracked without user intervention. The last sequence (Fig-
ure 10) contains many more challenges given the complex torso ro-
tations and arm swings. After introducing keyframing, our tracker
estimates the uncertain positions of arms and torso, and recov-
ers the motions of the arms even though they are totally occluded
in the silhouettes. For this sequence, six keyframes were used.
More reconstruction results can be found at http://www.cse.ohio-
state.edu/˜chenyis/research/motion/index.htm.
The time expended for analysis is about 3 sec. per frame for sim-
ple sequences and about 5 sec. per frame for difficult ones, when
executed on a Pentium IV PC with a 2 GHz CPU and 512MB of
memory. It should be noted that [3] reported times of 7 to 12 sec per

Figure 7: Analyzing the source of error appeared at two peaks in the
third plot of Figure 6: A,D and E are from deformation of clothes,
B is from motion blur effect, and C is from the inherent difference
between model and real image.
frame for similar operations. We use w = 1.0 from Eq. 3, we = 20
from Eq. 4, wl = 20 from Eq. 5 and wal = 20 from Eq. 6 for the
cost function for all motion synthesis.
7 C ONCLUSION AND F UTURE W ORK
We presented a simple and robust technique to reconstruct 3D mo-
tion parameters of a human model using image silhouettes. A novel
cost metric called core-weighted XOR was introduced and consis-
tently used for the automatic alignment, shape fitting and motion
tracking. The computation time of the cost function is directly re-
lated to the overall performance of our method. Currently our im-
plementation does not exploit any hardware acceleration. In the fu-
ture, we intend to accelerate the weighted-XOR computation using
features of a modern graphics hardware. Our work is closest to that
of Sminchisescu [20] and that of DiFranco [5]. We compare favor-
ably in terms of generality to the results they report. By detecting
more human features like head, arms, hands, legs and feet, we can
further improve the correctness of registering our model with the
all the images and recovering the 3d poses.
One of the biggest challenges in 3D motion reconstruction from
a single silhouette image is the inherent ambiguity caused by self-
occlusion of different body parts. Usually internal edge information
can be used to solve the ambiguity at certain degree. However,
spurious edges caused by shadows and/or mostly by the patterns of
clothes can result in incorrect reconstruction. Finally, we would like
to study how we can exploit various spatial and temporal features
of silhouette image sequence to infer the correct motion of self-
occluded part in future work.
ACKNOWLEDGEMENTS
The authors would like to thank the Advanced Computing Cen-
ter for Art and Design at Ohio State University for their support,
specifically for the use of their motion capture lab and software en-
vironment. The authors would also like to thank the folks at CMU
for making available their motion captured database. The database
was created with funding from NSF EIA-0196217. This work was
supported, in part, by U.S. National Science Foundation Grant ITR-
IIS-0428249 and by the Secure Knowledge Management Program,
Air Force Research Laboratory, Information Directorate, Wright-
Patterson AFB.
R EFERENCES
[1] A.F. Bobick and J.W. Davis. The Representation and Recognition of
Action using Temporal Templates. In IEEE Transactions on Pattern
Analysis and Machine Intelligence, volume 23, No.3, pages 257–267,
2001.
[2] C. Bregler and J. Malik. Tracking People with Twists and Exponential
Maps. In Proceedings of IEEE CVPR, pages 8–15, 1998.
[3] J. Carranza, C. Theobalt, M. Magnor, and H-P Seidel. Free-Viewpoint
Video of Human Actors. In Proceedings of SIGGRAPH 2003, volume
22, No.3, pages 569–577, 2003.
[4] J. Deutscher, A. Blake, and I. Reid. Articulated Body Motion Cap-
ture by Annealed Particle Filtering. In Proceedings of IEEE CVPR,
volume 2, pages 126–133, 2000.
[5] D.E. DiFranco, T. Cham, and J.M. Rehg. Reconstruction of 3-D Fig-
ure Motion from 2-D Correspondences. In Proc. Conf. Computer Vi-
sion and Pattern Recognition, pages 307–314, 2001.
[6] M. Gleicher and N. Ferrier. Evaluating Video-Based Motion Capture.
In Proceedings of Computer Animation, pages 75–80, 2002.
[7] I. Haritaoglu, D. Harwood, and L. Davis. Ghost: A Human Body Part
Labeling System Using Silhouettes. In International Conference on
Pattern Recognition, volume 1, pages 77–82, 1998.
[8] J.K. Hodgins. Carnegie Mellon University Graphics Lab Motion Cap-
ture Database, http://mocap.cs.cmu.edu/.
[9] T. Horprasert, D. Harwood, and L.S. Davis. A statistical approach
for real-time robust background subtraction and shadow detection. In
IEEE Frame-Rate Workshop, 1999.
[10] I. Kakadiaris and D. Metaxas. Model-Based Estimation of 3D Hu-
man Motion. In IEEE Transactions on Pattern Analysis and Machine
Intelligence, volume 22, December 2000.
[11] J. Lee, J. Chai, P.S.A. Reitsma, J.K. Hodgins, and N.S. Pollard. In-
teractive Control of Avatars Animated with Human Motion Data. In
Proceedings of SIGGRAPH 2002, Computer Graphics Proceedings,
Annual Conference Series, pages 491–500. ACM, ACM Press / ACM
SIGGRAPH, 2002.
[12] M.W. Lee, I. Cohen, and S.K. Jung. Particle Filter with Analytical
Inference for Human Body Tracking. In IEEE Workshop on Motion
and Video Computing, 2002.
[13] H.P.A. Lensch, W. Heidrich, and H. Seidel. Automated Texture Regis-
tration and Stitching for Real World Models. In Proceedings of Pacific
Graphics 2000, 2000.
[14] M. Leventon and W. Freeman. Bayesian estimation of 3-d human
motion from an image sequence. In Technical Report TR-98-06, Mit-
subishi Electric Research Laboratory, Cambridge, MA, 1998.
[15] T.B. Moeslund and E. Granum. A Survey of Computer Vision-Based
Human Motion Capture. In Computer Vision and Image Understand-
ing, 81(3), pages 231–268, 2001.
[16] V. Pavlovic, J.M. Rehg, T. Cham, and K.P. Murphy. A Dynamic
Bayesian Network Approach to Figure Tracking Using Learned Dy-
namic Models. In Intl. Conf. on Computer Vision, pages 94–101, 1999.
[17] W.H. Press, B.P. Flannery, S.A. Teukolosky, and W.T. Vetterling. Nu-
merical Recipes in C: The Art of Scientific Computing. Cambridge
University Press, New York, 1988.
[18] L. Ren, G. Shakhnarovich, J. Hodgins, H. Pfister, and P. Viola. Learn-
ing Silhouette Features for Control of Human Motion. In Proceedings
of the SIGGRAPH 2004 Conference on Sketches & Applications, Au-
gust 2004.
[19] C. Sminchisescu. Consistency and Coupling in Human Model Like-
lihoods. In IEEE International Conference on Automatic Face and
Gesture Recognition, pages 27–32, 2002.
[20] C. Sminchisescu and B. Triggs. Kinematic Jump Processes For
Monocular 3D Human Tracking. In Proc. Conf. Computer Vision and
Pattern Recognition, pages 69–76, 2003.
[21] Cristian Sminchisescu and Bill Triggs. Estimating Articulated Human
Motion with Covariance Scaled Sampling. International Journal of
Robotics Research, 2003.
[22] C.R. Wren, A. Azarbayejani, T. Darell, and A.P. Pentland. Pfinder:
Real-Time Tracking of the Human Body. In Transactions on Pattern
Analysis and Machine Intelligence, 19(7), 1997.
Figure 8: Tracking a dancer: displaying representative frames. The top row shows the footage, the middle row is the matching between
silhouette (white area) and our model (colored area), and the bottom row is the animation after rendering. The limb stretching term is used
here.

Figure 9: Tracking a badminton player: displaying representative frames. The limb stretching term for legs is used here.

Figure 10: Tracking an actor: displaying representative frames. We employ the keyframe technique on the arms to reconstruct this sequence.