Physical Face Cloning

Bernd Bickel
1
Peter Kaufmann
1
M´ elina Skouras
1,2
Bernhard Thomaszewski
1,2
Derek Bradley
1
Thabo Beeler
1
Phil Jackson
3
Steve Marschner
4
Wojciech Matusik
1
Markus Gross
1,2
1
Disney Research Zurich
2
ETH Zurich
3
Walt Disney Imagineering
4
Cornell University
Figure 1: Left: photograph and scanned 3D geometry of a human face. We use a physics-based optimization process to design the geometry
of a synthetic skin in order to best match given target expressions. Right: final animatronic figure with fabricated skin.
Abstract
We propose a complete process for designing, simulating, and fab-
ricating synthetic skin for an animatronics character that mimics the
face of a given subject and its expressions. The process starts with
measuring the elastic properties of a material used to manufacture
synthetic soft tissue. Given these measurements we use physics-
based simulation to predict the behavior of a face when it is driven
by the underlying robotic actuation. Next, we capture 3D facial ex-
pressions for a given target subject. As the key component of our
process, we present a novel optimization scheme that determines
the shape of the synthetic skin as well as the actuation parameters
that provide the best match to the target expressions. We demon-
strate this computational skin design by physically cloning a real
human face onto an animatronics figure.
Keywords: animatronics, physics-based simulation, computa-
tional material design, facial animation, optimization.
Links: DL PDF WEB VIDEO
1 Introduction
We are naturally intrigued by the prospect of creating virtual hu-
mans in the likeness of ourselves — and it is not far-fetched to
say that this is also a driving force for computer graphics research.
While the latter strives to photorealistically recreate human charac-
ters on a computer screen, animatronics aims at creating physical
robot characters that move and look like real humans. Both fields
share many similarities; however, animatronics faces harder chal-
lenges since it has to deal with the physical constraints of real ma-
terials and actuators in addition to the virtual modeling component.
The human body consists of articulated rigid structures (bones) and
soft tissue (e.g., flesh and skin). Therefore, it seems natural for an-
imatronic characters to have a rigid articulated base and synthetic
soft tissue. Many impressive characters have been created in this
spirit, e.g., those in Disney World’s Hall of Presidents or “Gemi-
noids” [?], androids that closely resemble human beings. However,
creating such figures is still a difficult and labor-intensive process
requiring manual work of skilled animators, material designers, and
mechanical engineers. Owing to its expressive power, the human
face is probably the most challenging part in this context. An ani-
matronic character has to produce a vast range of facial expressions,
each having different deformations and wrinkles. Manually design-
ing the shape and material properties of a single skin that is able to
achieve all these targets is clearly a formidable task.
The goal of this work is to automate this process, to increase the
realism of the resulting character and, ultimately, to create an an-
imatronic face that closely resembles a given human subject. In
order to accomplish this task, we capitalize on recent developments
from three areas in computer graphics: facial performance capture,
physics-based simulation, and fabrication-oriented material design.
Building on these foundations, we present a method of physical face
cloning — a novel process for computational modeling, optimiza-
tion and fabrication of synthetic skin for animatronic characters.
More specifically, our process comprises the following steps. First,
we capture elastic material properties for a range of possible syn-
thetic skin materials using a custom measurement system. We sub-
sequently capture a collection of different expressions for a given
target human face. As the central part of our pipeline, we then op-
timize the geometry of the skin and the actuation parameters of the
underlying animatronics device to provide the best match to the tar-
get human face. Although our method applies quite generally to a
broad range of possible animatronic devices, we validate the whole
process in this paper by fabricating synthetic silicone skins for a
specific articulated robot head. These skins are then animated, and
the resulting shapes are compared both to the optimized model and
to the real face.
The contributions of the research presented in this paper can be
summarized as follows:
• A method to optimize for synthetic skin geometry and actua-
tion parameters that provide the best match to a collection of
given target expressions,
• A complete process for automating the physical reproduction
of a human face on an animatronics device, including data
acquisition, physical simulation, optimization, fabrication and
validation,
• A comprehensive analysis and experimental validation of the
process and its individual steps.
2 Related Work
Physical face cloning is a complex task that intersects with numer-
ous research areas from computer graphics, applied mathematics,
and material sciences. Although we are not aware of existing work
to address this problem in its entirety, the individual components of
our system have diverse connections to previous research.
Face Capture. Owing to the expressive power of the human face
and our ability to infer a burst of information from the slightest mo-
tion, capturing, synthesizing and transferring facial expressions are
important research topics in computer graphics. There exists a mul-
titude of methods for capturing facial expressions, including tradi-
tional marker-based motion capture [?; ?], structured light systems
[?; ?], and passive markerless approaches [?; ?; ?]. Our acquisition
system uses the recent method of Beeler et al. [?] to capture short
performance sequences with high-resolution geometry and dense
temporal correspondences.
Simulation. Introduced to the graphics community by Terzopou-
los et al [?], it has become common to model elastically deformable
materials using continuum mechanics and finite elements. There
exists a plethora of dedicated bio-mechanical models for different
parts of the human body such as the face [?; ?; ?], the hand [?; ?],
the torso [?], various organs [?], or even the entire upper body [?].
Similarly, a large body of literature is concerned with modeling of
biological soft tissue (see, e.g., the work by Teran et al. [?] and
references therein). We do not strive to reproduce the mechanical
properties of biological tissue, but rather take an output-oriented
approach: we fabricate synthetic skin using silicone rubbers, which
are adequately modeled as elastic materials with isotropic and ho-
mogeneous properties. More specifically, we rely on a compress-
ible neo-Hookean material model and, on the computational side,
employ a finite element approach centered around linear tetrahedral
elements.
Measuring and Fitting Material Parameters. Closely related to
the choice of a constitutive law is the problem of finding a set of
parameters that best approximates a given reference model. This
reference data can originate either from an accurate computational
model [?; ?; ?] or, as in our case, from measurements of real-
world objects. In the context of computer graphics, several meth-
ods have been described for fitting measured data to computational
models, including linear-elastic materials [?], nonlinear viscoelas-
tic soft tissue [?], or nonlinear heterogeneous materials [?]. Pai
et al. [?] describe a comprehensive system for capturing interac-
tions with real-world objects, including deformation response, sur-
face roughness and contact sounds. Our approach for determining
parameters of different silicone samples goes along similar lines:
we use a vision-based acquisition system to capture dense surface
displacement fields and determine optimal material parameters by
minimizing the displacement error of a corresponding finite ele-
ment approximation.
Fabrication-oriented Design. As a natural step to follow acqui-
sition and data-driven simulation, fabrication-oriented design is a
recent trend that is seeing increasing attention in computer graph-
ics. Several methods have been presented in this context, targeting
the reproduction of geometric features of papercraft [?] or plush
toys [?], general appearance properties [?; ?; ?], and also the elastic
properties of deformable media [?]. Our approach for fabrication-
oriented material design bears some similarities to this last refer-
ence, but there is also a crucial difference: whereas Bickel et al.
determine spatially varying material properties for fixed geometry,
we fabricate synthetic skins with uniform material properties and
optimize their shape.
Shape Optimization. Engineering applications often require the
shape of a structure to be optimal with respect to given conditions.
Examples of such shape optimization problems include the com-
putation of aerodynamic profiles, the design of reflectors for lumi-
naires or the optimization of material interfaces. Such problems are
typically cast as constrained optimization problems and solved nu-
merically with finite elements (see, e.g., the textbook by Bucur and
Buttazzo [?] for an introduction). We optimize the shape of the
synthetic skin by minimizing an elastic energy with respect to rest
state positions, which is similar in spirit to methods for variational
shape adaptation [?]. But since it is generally difficult to control
the adaptation of the undeformed configuration, we use a dedicated
parametrization approach combined with a moving least squares in-
terpolation technique that allows us to modify the shape only in its
thickness direction.
Robotics & Animatronics. The majority of robots consist of
only rigid mechanical structures. However, robots that are cov-
ered with soft-tissue materials such as rubbers or silicones have
been built as well. Hara et al. [?] was one of the first group
to build an anthropomorphic head to study human-robot interac-
tion. The face skin was made of silicone and it was driven by
SMA (shape memory alloy) actuators. The head was able to re-
produce six different expressions such as surprised, scared, an-
gry, disgusted, sad and happy. Android research, which involves
studying human-computer interaction, has been carried out by In-
telligent Research Laboratory at Osaka University. The group has
built a number of robotic characters with increasing capabilities.
Their recent humanoid robot Geminoid HI-1 [?] as well as it’s
successor Geminoid-DK have the realistic appearance of an adult
man. These robots also have silicone-based skin driven by actua-
tors placed underneath. KAIST and Hanson Robotics have devel-
oped a robot called Albert HUBO [?]. The robot’s head resembles
Albert Einstein and its underlying electro-mechanical actuation has
35 degrees of freedom. The skin is made of a sponge-like elastomer
which is very soft and requires very little force to deform. In con-
trast to our approach, the design process for all these robotic skins
is mainly manual. Shape variations are modeled by an artist, and
the implications of changes to the shape have to be tested in a time
consuming trial and error manner.
3 Overview
Our approach to physical face cloning comprises modeling, simula-
tion, design, and fabrication of synthetic soft tissue. The individual
steps of our processing pipeline are illustrated in Fig. 2.
We start by capturing facial expressions of a human subject using
an optical performance capture system that offers high-resolution
3D reconstructions including pores and wrinkles, as well as robust
temporal correspondence (Sec. 4). This data provides information
about the deformation behavior of the subject’s skin. In addition,
Face Meshes
Animatronic
Head
Skin Attachement
Paths
f(p
i
)=
Material
Sample
Material
Parameters
Acquisition Optimization
Actuation
Parameters
Material
Thickness
Fabrication
Final Skin
3D Printed
Mold
Figure 2: Physical face cloning pipeline.
we also determine the operational range of the animatronic head,
which consists of a set of skin attachment links actuated by motors.
We attach to each moving link a marker, which allows us to sample
a mapping of actuation parameters pact to 3D location and orien-
tation of each link in space. Interpolating these samples yields a
continuous function m(pact) that describes the path of the skin at-
tachment links (Sec. 7). Given the acquired example shapes of the
human face and the description of the animatronic head, the goal is
now to design synthetic soft tissue that matches the example shapes
as closely as possible.
Our base material for the synthetic skin is silicone, which of-
fers a wide range of stiffness that can be controlled by adjust-
ing the concentration of plasticizer in the compound. In order to
determine material parameters for the skin simulator, we numeri-
cally fit our computational model to experimentally acquired force-
displacement samples of materials with different stiffnesses. We
model and simulate the deformation behavior using a non-linear fi-
nite element method in combination with a neo-Hookean material
(Sec. 5). The forward simulation allows accurate prediction of the
deformed shape of a given synthetic skin and the resulting forces
induced by the moving skin attachment links.
In order to design a synthetic skin that best matches a desired tar-
get, we introduce a novel optimization process. While the outer
surface of the skin in the undeformed configuration is given by a
3D scan of the neutral target pose, we propose to vary the inner
surface (thickness) of the skin to achieve spatially varying stiffness
and deformation behavior. Furthermore, we also optimize the ac-
tuation parameters of the animatronic device to find the parameters
that best resemble the individual target expressions. All these de-
grees of freedom are handled in one uniform optimization frame-
work (Sec. 6).
We validate our simulation and optimization approach by fabricat-
ing various silicone objects using injection molding (Sec. 8). The
molds are created using a rapid prototyping 3D printer. We then
compare predicted simulation results and real-world behavior for
different deformation scenarios (Sec. 9).
4 Measurements and Acquisition
We rely on real-world data for modeling facial geometry and ex-
pressions as well as material behavior. The following paragraphs
briefly describe our techniques for data acquisition and integration.
Face Scanning. When cloning a human face, matching the shape
of the face and how the skin deforms under different expressions is
one of the main challenges. We must reconstruct various 3Dexpres-
sions of the subject to be cloned, in order to seed the deformable
skin optimization. Our approach is to capture a short performance
of the target subject, leveraging the high resolution geometry acqui-
sition method of Beeler et al. [?]. The result is a detailed, compati-
ble mesh sequence with explicit temporal correspondence, allowing
us to analyze the deformation of a face under various expressions,
down to the level of individual wrinkles. In our case, we chose 8
expressive poses from the sequence for optimizing the skin param-
eters.
Measuring & Fitting Material Parameters. Silicone rubber is an
ideal base material for fabricating synthetic skin since it is easy to
process and offers a vast range of stretch resistance. The anima-
tronics device can only exert a limited force through its actuators,
which imposes an upper bound on the net stiffness of the synthetic
skin. Hence, we have to adjust the silicone composition such that
the net stiffness of the synthetic skin complies to these constraints.
To accomplish this task, we first measure the force-deformation be-
havior of a collection of silicone samples with different amounts of
plasticizer. We then perform numerical optimization in order to de-
termine the parameters of our computational model that best match
the experimental data.
The measurement proceeds by pulling on small samples with con-
trolled forces and capturing the resulting deformation over time us-
ing stereo reconstruction [?]. With a set of applied loads and re-
sulting surface displacements determined, we optimize the material
parameters of our finite element solver in order to best approxi-
mate the measured stress-strain behavior. To this end, we start by
defining an objective function O(p) =

i
| (˜ xi −xi(p)) |
2
that
measures the difference between simulated surface positions xi(p)
and their closest corresponding point of the captured surface ˜ xi.
The vector p holds the physical parameters of the material model,
which can be identified with Young’s modulus and Poisson’s ra-
tio (see Sec. 5). Given an initial guess p0 we iteratively compute
updated parameters in a simulated annealing process. In each itera-
tion, we compute the finite element solution x(pi) with the current
parameters, evaluate the objective function O(pi) and determine
updated parameters pi+1. This process is repeated until the objec-
tive function signals convergence.
As a validation for this numerical-experimental material fitting pro-
cess, we compare the stress-strain curves obtained from a standard
testing procedure (ISO 37) to those obtained from a virtual coun-
terpart of the same experiment. As can be seen from Fig. 4, we
are able to match the real material behavior quite closely for defor-
mations in the range of [−20%, +50%]. Considering the operating
range of our animatronics device, this provides us with sufficiently
large safety margins.
5 Physical Simulation of Synthetic Skin
As a central part of the face cloning pipeline, we need to have an
accurate computational model for simulating deformations of a syn-
thetic skin. Our approach is to model skin as a hyperelastic isotropic
solid. Since we have to account for large rotations, stretching, and
compression, we resort to finite-deformation continuum mechan-
ics. Here we only give a brief overview and refer the reader to the
textbook by Bonet and Wood [?] for a comprehensive exposition.
Let X and x denote smooth functions describing the position of
the skin in its undeformed and deformed state, respectively. Fur-
ther, let ϕ : Ω → IR
3
denote the mapping that transforms mate-
rial points from the undeformed to the deformed configuration as
Figure 3: Illustration of thickness optimization. (a) Target surface
(b) undeformed skin patch (c) unoptimized skin patch deformed by
vertical load (d) deformed skin patch with optimized thickness.
x = ϕ(X). The deformation of the skin at each point can be char-
acterized by the deformation gradient F or, alternatively, the right
Cauchy-Green tensor C which are defined as
F =
∂ϕ
∂X
, respectively C = F
T
F .
For a hyperelastic material, the energy of the solid depends only on
its current state of deformation, which is described in a rotation-
and frame-invariant manner through the tensor C. We denote the
corresponding energy density function by Ψ = Ψ(C). The rela-
tion between Ψ and C is described by material models, which are
discussed next.
5.1 Material Model
Animating expressions on a virtual face generally induces both fi-
nite stretching and compression of the skin. While the simple St.
Venant-Kirchhoff solid cannot be used in this setting [?], the non-
linear neo-Hookean, Mooney-Rivlin, and Ogden material models
are well suited for rubber-like substances such as silicone. Being
the simplest representative of this class, the strain energy density of
the compressible neo-Hookean material is given as
Ψ(C) =
µ
2
(J

2
3
tr(C) −3) −
κ
2
(J −1)
2
, (1)
where J = det F and µ, κ are shear and bulk moduli, which are
related to the familiar Young’s modulus E and Poisson’s ratio ν as
E =
9µκ
3κ +µ
, respectively ν =
3κ −2µ
2(3κ +µ)
. (2)
The neo-Hookean material offers only two parameters, but for the
range of deformations considered here, we found this model to be
sufficiently accurate for obtaining good correspondence with the
measured data. Having settled upon this material model, we de-
termine the model parameters µ and κ for a given material sample
using the numerical-experimental technique described in the previ-
ous section. Fig. 4 compares the force-strain curve for a measured
and simulated material.
5.2 Discretization
The optimization process described in the next section requires the
minimization of the elastic energy subject to boundary conditions.
To this end, we discretize (1) in space using tetrahedral finite ele-
ments. Let xa ∈ IR
3
, 1 ≤ a ≤ n denote deformed positions of a
tetrahedral mesh. We will use the superscript e to identify quanti-
ties pertaining to a given tetrahedral element. In particuar, we let
X
e
∈ IR
12
and x
e
∈ IR
12
denote position vectors for a given ele-
ment in its undeformed and deformed state, respectively.
Figure 4: Force-strain curve of real (black) and simulated (red)
material.
We proceed by approximating the configuration mapping as ϕ ≈

a
xaNa where Na are piece-wise linear basis functions associ-
ated with the nodes of the mesh. The discrete deformation gradient
for each element e is then expressed as
F
e
= F(x
e
, X
e
) =
4

a=1
x
e
a
_
∂N
e
a
∂X
_
T
, (3)
where we emphasize the fact that for each element F
e
is a function
of its position in both undeformed and deformed configuration. All
other discrete quantities required for computing the elastic energy
of a deformed element, i.e. tr(C
e
) and J
e
, follow directly from
(3). Furthermore, we note that the deformation gradient is constant
within each element since N
e
a
are piece-wise linear functions. With
this relation established, the discrete elastic energy of a deformed
element W
e
can be expressed as
W
e
(x
e
, X
e
) =
_

e
Ψ(F
e
) = Ψ(F
e
)V
e
(X
e
) , (4)
where Ω
e
is the element’s parameter domain and V
e
its unde-
formed volume. From now on, let x = (x
T
1
, . . . , x
T
n
)
T
and
X = (X
T
1
, . . . , X
T
n
)
T
denote vectors containing all deformed and
undeformed positions respectively. Summing up all elemental con-
tributions, we obtain the total energy for a deformed configuration
as
W(x, X) =

e
W
e
(x
e
, X
e
) , (5)
We can now use this energy in a static equilibrium problem in or-
der to compute the deformation of the skin in response to actuator
placements, which translate into a set of position constraints. The
deformed configuration of the skin is then determined as the mini-
mum of the total energy
x = argmin
x
(W(x, X) +Wext(x)) , (6)
where Wext summarizes external contributions to the potential, e.g.
due to gravity. Requiring the gradient of the energy to vanish at
equilibrium leads to a set of nonlinear equations, which are solved
in the standard way using Newton iterations with incremental load-
ing and line search for convergence control.
By solving (6), we can simulate skin deformation in response to
arbitrary actuator placements.
6 Optimization
Recall that our goal is to fabricate a synthetic skin which, when
deformed by the underlying electromechanical base, matches the
target expressions as closely as possible. In order to achieve this
requirement, we optimize the shape of the skin as well as the actu-
ation parameters of the base. This process is subject to a number of
constraints. We cannot change the outside surface of the skin. We
also cannot modify the attachment points of the actuators. Further-
more, the motion of the actuators is physically restricted.
In order to achieve the best results given these constraints, we have
to expose as many degrees of freedom as possible to the optimiza-
tion process. To this end, we optimize with respect to two sets of
parameters. First, we optimize the actuation parameters of the an-
imatronic device for every target expression. Second, we change
the thickness of the skin by modifying its shape on the inside – we
are free to do so everywhere except at the attachment points and in
their immediate vicinity. Practically, this allows the skin to produce
bulges and wrinkles where they did not appear before. We start by
describing the general optimization framework and then how we
apply it to the individual parameter sets.
6.1 Generic Optimization Framework
Let the function
ˆ
W(ˆ x, p) denote the total energy of our physical
system, including internal deformation energies and external con-
tributions to the potential. p is a vector of generic parameters for
which we want to optimize. ˆ x contains a subset of the deformed
positions x of the physical simulation. All other values of x are
either constant (constrained) or computable from p. In the most
general case, both the undeformed and deformed positions of our
simulation mesh are allowed to depend on p, and we can write
ˆ
W(ˆ x, p) = W(x(ˆ x, p), X(p)) +Wext(x(ˆ x, p)). (7)
For fixed parameter values p, the physical system will assume the
equilibrium state
ˆ xeq(p) = argmin
ˆ x
_
ˆ
W(ˆ x, p)
_
(8)
minimizing the total energy of the system, leading to the necessary
condition of a vanishing residual force

ˆ
W
∂ˆ x
¸
¸
¸
¸
ˆ xeq,p
= 0. (9)
The goal of the optimization will be to find the optimal parameter
values p such that the positions x(ˆ xeq(p), p) match a desired tar-
get configuration as closely as possible. This is measured in terms
of a matching energy E
match
(x). We additionally regularize the
parameters using a regularization energy Ereg(p).
We solve this as a minimization problem with respect to ˆ x and p,
with the following objective function:
E(ˆ x, p) =
1
2
γ
¸
¸
¸
¸
¸
¸
¸
¸

ˆ
W
∂ˆ x
¸
¸
¸
¸
ˆ x,p
¸
¸
¸
¸
¸
¸
¸
¸
2
+E
match
(x(ˆ x, p)) +Ereg(p) (10)
The first term penalizes the violation of condition (9), ensuring with
a sufficiently high penalty γ that the resulting ˆ x is close to a physi-
cal solution with vanishing residual forces.
Matching Energy. The matching energy measures how well a
configuration x is in agreement with what we want the object to
look like. To this end, we embed a set of points in the simu-
lation domain and deform them according to the mapping ϕ. In
our implementation, we use a subset of points from high-resolution
3D scans (Sec. 4). Due to our choice of basis functions, the de-
formed position of a point can be computed as a linear combination
of four nodal positions. We then compute the matching energy as
the squared distance between the deformed points and their desired
target positions q,
E
match
(x) =
1
2
||Sx −q||
2
(11)
where the matrix S contains the weights for the computation of the
deformed points from x.
Numerical Optimization. We minimize (10) using a Newton
method. For this we need to compute its first and second deriva-
tives with respect to ˆ x and p. While these computations are trivial
for our particular choices of matching and regularization energies,
we compute the second derivatives of the first term in (10) only ap-
proximately, ignoring the third order derivatives of W that would
result from applying the chain rule. In each Newton step, we use a
sparse direct solver [?] to solve the linear system
H
_
∆ˆ x
∆p
_
= −f (12)
for the increments ∆ˆ x and ∆p, where f is the vector of first deriva-
tives of E and H contains its second derivatives. We employ line
search to make sure the energy decreases in each Newton step. It
can happen that the matrix H becomes indefinite, a condition we
can easily detect by the direct solver not being able to factorize H,
in which case we proceed as suggested in Nocedal and Wright [?]
by adding a multiple of the identity to H.
6.2 Thickness Optimization
The goal of the thickness optimization is to modify the local thick-
ness of the skin in such way that when the mechanical actuators of
the robot are set to the values corresponding to a particular pose,
the resulting deformation of the skin matches the pose’s target po-
sitions q as closely as possible. In the physical simulation, the actu-
ator settings result in hard positional constraints that can directly be
applied to the corresponding deformed positions. The parameters
p
thk
determine the thickness distribution in the undeformed config-
uration without directly affecting the deformed positions, thus we
can write
ˆ
W(ˆ x, p
thk
) = W(ˆ x, X(p
thk
)) +Wext(ˆ x) (13)
and minimize (10) to find the parameter values for the thickness
distribution that best match the given pose.
Parameterizing the Undeformed State. Although the skin
clearly exhibits a thickness direction, this information is not explic-
itly available from the 3D geometry such that we have to infer it
first. We start by computing a low-pass filtered version of the skin’s
outer surface and construct a parameterization r : IR
2
→IR
3
using
standard uv-mapping software. A roughly area-preserving map is
desirable for obtaining unbiased sample distributions, but this is not
a critical aspect in this context. The undeformed geometry of the
skin can now be described as X = r(u, v)+hn(u, v), where u and
v are surface coordinates and the thickness parameter h runs along
the surface’s normal direction n. Having defined the thickness in
this way, we can now proceed to its optimization.
Instead of working directly on the undeformed positions, we define
a warping heightfield on the surface in order to smoothly deform
space along its normal direction. We construct a smooth warp-
ing field w(u, v) from a sparse set of height samples αi at po-
sitions (ui, vi)
T
using moving least squares (MLS) interpolation
[?]. Fig. 5 illustrates the concepts of parametrization and MLS in-
terpolation, while Fig. 6 shows some shape variations that can be
Figure 5: Left: parametrization of the geometry and placement
of MLS samples; Right: MLS warping field shown as surface with
isoparametric lines for illustration.
obtained for a simple one-dimensional example. To control the res-
olution of the warping heightfield, we adaptively place the locations
of the height samples in uv space based on a user-provided density
map [?].
Evaluating the MLS warping field at an arbitrary parameter point
(u, v) amounts to fitting an affine transformation that minimizes
the weighted least squares error

i
θ(||(u, v)
T
−(ui, vi)
T
||)||a(u, v)
T
(1, ui, vi) −αi||
2
, (14)
where θ(x) is a weighting kernel and a(u, v) ∈ IR
3
are the local
coefficients of the interpolation function. The latter are determined
by solving the normal equations for (14), which can be done effi-
ciently using precomputations.
Finally, we gather the parameters αi of all sample points into p
thk
and compute optimal height values by minimizing (10). Note
that the MLS interpolation results in a linear mapping between
undeformed positions X and parameters p
thk
, thus the matrix
∂X/∂p
thk
is constant and can be precomputed. Once the opti-
mal αi are determined, the final rest configuration is obtained in
the following way: for each point Xa of the original mesh we first
retrieve its parameter values (ua, va, ha), evaluate the MLS warp-
ing field to obtain wa = w(ua, va) and finally compute the warped
positions as X

a
= r(ua, va) +wahan(ua, va).
Regularization. We need the undeformed mesh with nodal posi-
tions X(p
thk
) to be a valid tetrahedral mesh where every element
has positive volume. While there are different ways to formulate
and enforce this constraint, we found it most practical to reuse the
deformation energy function W, abusing X(p
thk
) as the deformed
configuration and the initial undeformed positions X(1) as the un-
deformed configuration:
Ereg(p
thk
) = γ
undefreg
W(X(p
thk
), X(1)) (15)
For any positive penalty value γ
undefreg
, this energy will tend to
infinity as any element’s volume approaches zero, effectively pre-
venting element inversions in the undeformed configuration.
Fabrication Constraints. Additionally, we add a quadratic en-
ergy term to Ereg that penalizes MLS weights exceeding a value
of one, or falling below a user-defined minimum thickness. This
guarantees a fabricatable skin that does not intersect the underlying
mechanical structure.
Multiple Pose Optimization. We want to optimize the unde-
formed configuration X(p
thk
) such that m > 1 poses can be re-
produced as closely as possible: for a set of actuator settings ai, the
given target positions qi should be matched as closely as possible.
In this case our optimization is slightly modified and we optimize
for all deformed positions ˆ x1, . . . , ˆ xm and p
thk
at once. We have
Figure 6: Illustration of MLS interpolation using 7 sample points.
(a) Input geometry (b)–(d) examples of warped geometry.
m matching functions E
i
match
and the objective function can be
written as
E(ˆ x, p
thk
) =
m

i=1
1
2
γ
¸
¸
¸
¸
¸
¸
¸
¸

ˆ
W
∂ˆ x
¸
¸
¸
¸
ˆ x
i
,p
thk
¸
¸
¸
¸
¸
¸
¸
¸
2
(16)
+
m

i=1
E
i
match
(x(ˆ xi, p
thk
)) (17)
+Ereg(p
thk
). (18)
We initialize the optimization with values ˆ xi computed from a per-
pose forward simulation with incremental loading, and use p
thk
=
1 for the MLS parameters.
6.3 Actuation Parameter Optimization
The goal of the actuation parameter optimization is to find the best
values for controlling the mechanical actuators such that the result-
ing deformed skin matches a given target pose as closely as possi-
ble. In this setting, the parameters pact of the optimization define
the positional constraints to apply to the nodal positions x. In the
objective function (10) we use
ˆ
W(ˆ x, pact) = W(x(ˆ x, pact), X) +Wext(x(ˆ x, pact)). (19)
Parameterization. The mapping from unconstrained nodal posi-
tions ˆ x and parameters pact to deformed positions x is
xi =
_
ˆ xi if node i unconstrained
Ma(pact)Xi if node i constrained by actuator a.
(20)
For the sake of simplicity, we linearize the transformation matrix
Ma(pact) of actuator a around the current parameter values pact
in each step of the optimization.
Regularization. Similar to the thickness optimization, we use the
regularization energy Ereg to keep the actuator parameter values
within the range of physically feasible values.
7 Animatronic Head
To actuate the silicone skin we use a proprietary ani-
matronic head developed by Walt Disney Imagineering.
It is driven by electric motors and features
a = 13 parameters pact ∈ IR
a
to con-
trol the movement of the attachment links
indicated in red in the figure on the left.
The length of the red arrows illustrates
the range of potential motion. Further-
more, there are several rigid attachment
links that constrain the movement of the
skin. We refer to the video for a visual-
ization of the actuation level of each ac-
tuation parameter for a facial animation
sequence, providing an intuition of which
expressions are inside or at the border of
the gamut of reproducible poses. We treat the underlying mechan-
ics and controlling mechanism as a black box, so in theory our op-
timization algorithm can be used with any animatronics device.
To determine the operating range and control parameters of the head
we attach a marker to each moving link and track its position under
a number of sample locations and orientations. Tracking is per-
formed using the same capture system as used for face scanning
(Sec. 4). The mapping m(pact) : IR
a
→ IR
l·6
from animation pa-
rameters pact to 3D locations and orientations of attachment points
is then specified by quadratically interpolating the samples. These
are then used as hard boundary constraints in our forward simula-
tion. To obtain the specific animation parameters for a given facial
expression, we solve the inverse problem as described in Sec. 6.3.
8 Fabrication
Our choice to use silicone is based on the fact that the stiffness can
be controlled accurately, the color can be adjusted, it is robust, and
the fabrication process is easy and safe. Silicone belongs to the
class of elastically deformable polymers with a high yield strength.
After having optimized the ideal skin geometry, we use liquid injec-
tion molding to fabricate the physical skin as shown in the accom-
panying video. Our mold consists of a core that resembles the inner
surface of the skin and six outer parts. Note that multiple outer parts
are required so that the skin can be removed without destroying the
mold. The mold itself is fabricated using a rapid prototyping 3D
printer. For our experiments, we used the two-part silicone GI-245
by Silicones Inc. After mixing it with the corresponding catalyst, it
has a liquid consistency and can be injected into the mold. Approx-
imately seven days of room-temperature curing is required until it
is ready to be used.
9 Results
In this section we present our results, starting with validation of our
pipeline and optimization process, and then showcasing our method
with an example of real physical face cloning for an animatronic
figure.
Validation Simulation. We validated our pipeline for measuring,
modeling, and simulating silicone materials. Fig. 4 shows a com-
parison of the force-strain curve of a simulated material to a mea-
sured sample. The measurements are acquired by running an uni-
axial stretch and compression test on a rectangular (55 × 5 × 2
mm — stretch) and cylindrical (50 mm diameter, 12 mm height
— compression) object. Furthermore, we validate the simulation
by comparing simulated deformations of our silicone skin with 3D
scanned poses of the actual animatronic figure in Fig. 9, showing
an average error of less than 1.35 mm.
Validation Optimization. Our optimization process computes
the volumetric shape of soft-tissue material under actuation such
that the surface matches the deformation of given target surfaces.
We validated the optimization process in simulation. We start by
generating ground truth data by randomly varying the thickness of
a uniformblock using the moving least squares warp as described in
Sec. 6.2. From the forward simulation of these objects we extract
target surfaces. We then used these target surfaces and the unde-
formed block as input into our optimization — see e.g. Fig. 7. In
(a) (b) (c) (d) (e)
Figure 7: Volumetric shape optimization. Starting with a volumet-
ric object (a) and a target surface (b), our optimization process
computes the thickness variation (c) such that the surface matches
the deformation of the target under actuation (d). Without optimiza-
tion (e), the target is not well approximated.
all experiments, our algorithm was able to exactly reproduce the
spatially-varying thickness of the ground truth objects that were
used to generate the inputs. As shown in Fig. 8, we also fabricated
a subset of these examples, and compared the deformation of the
predicted optimized shape to the real object. For these 95mm long
bars, the average errors between the deformed fabricated surface
and the goal shape are 0.44mm (top) and 0.38mm (bottom).
Face Cloning. We showcase our complete pipeline by physically
cloning a human face for an animatronic figure. We started by ac-
quiring more than 100 frames of an expressive performance of our
subject. We then aligned the neutral pose to a scan of the anima-
tronic head. Because the animatronic head has very pronounced
cheek bones, we slightly warp the acquired 3D geometry of the
human face to prevent intersections. Consequently, we apply the
same warping transformation to the captured geometry before do-
ing comparisons (such as the error plots in Fig. 10) with our results.
The gap between the surface of the neutral pose and the underlying
animatronics defines the initial unoptimized skin, which we repre-
sent as a tetrahedral mesh consisting of ≈ 63k elements. To better
represent important facial deformation features on the forehead, we
additionally use a high-resolution mesh of ≈ 157k elements to op-
timize this area. We then use our optimization process to obtain
animation parameters for the entire captured face performance se-
quence. As matching criteria we select small surface regions close
to the attachment links. In those regions, the surface thickness is
constrained and cannot be changed because the skin is connected
to the moving and static links. For all other regions we use our op-
timization process to achieve the desired deformation behavior by
varying the skin thickness. In our experiment, we used a subset of 8
expressive poses for thickness optimization. Fig. 10 illustrates the
unoptimized and optimized geometry and provides a side-by-side
comparison of the deformation behavior.
Finally, we also fabricated the optimized skin, using material with
Young’s modulus of E = 78kPa and Poisson’s ratio of ν = 0.47
in its cured state. Figure finishing included industry-standard tech-
niques for adding hair and facial stubbles and painting a texture
by a professional makeup artist. Fig. 9 shows our figure in action
and compares it to the human face and the simulation under vari-
ous facial expressions. We also scanned these poses of the anima-
tronic figure and provide error visualizations showing the distance
between the simulated surface and the scanned surface. As can be
seen from the error plots, the agreement between the shape pre-
dicted by our simulation and the actual observed shape of the actu-
ated skin is generally good, with less than 4mm separating the sur-
faces everywhere. This shows that our simulation is a good model
for the real synthetic face. The difference between the scans of the
real face and the simulation is larger, representing the limitations in
the range of motion of the robot head and the range of deformation
of the attached skin. Comparing the human face to the cloned syn-
thetic face, the large scale deformations and general expressions
are captured well, and the differences show where improvements
should be made in the physical system to enable the fitting methods
to achieve a better match. For further comparisons we refer to the
accompanied video.
10 Conclusion
In this paper we have described a process for computationally
guided design of an animatronic character composed of a soft-tissue
material and an electromechanical base. We compute a volumetric
shape of the soft-tissue material and a set of actuation forces that
can reproduce target deformations of an exterior character surface.
As an example, this process allows us to construct an animatronic
head that approximates the shape and behavior of a real person
whose expressions are acquired using a 3D scanning system. The
core requirement of this process is a physically based simulation
framework that lets us accurately predict the behavior of the soft
tissue when subject to external forces. Our experimental validation
shows that using this process provides a principled way to design
soft-tissue animatronic characters.
Limitations and Future Work. We believe that our computation-
ally guided process for designing and fabricating animatronic char-
acters will serve as a blueprint for how future robots with soft tissue
should be built. Because this work is the first in the area, it has a
number of limitations that should be addressed in future research.
First, in our current framework the attachment of the actuators is
fixed to specific places underneath the character’s skin. Relaxing
this constraint would likely give us more degrees of freedom, e.g.,
the rest shape of the soft tissue would not need to be the same as
the rest shape of the electromechanical base. Second, we currently
only use a single soft tissue material to produce a synthetic skin. We
believe we can extend the range of possible deformations by using
multi-layered materials similarly to Bickel et al. [?]. Moreover, it
is important to note that the motions of current animatronic charac-
ters are still limited as they are less expressive than most humans.
Thus, it is important to develop retargeting algorithms that can map
the expressions of a target human to the physical constraints of the
electromechanical base and materials used. In this paper, we have
not addressed the appearance aspect of the synthetic skin tissue in-
cluding reflectance, subsurface scattering, and hair. This is also a
very interesting direction for future work and we envision that sim-
ilar computationally guided processes could be developed to solve
these problems. Finally, while in this work we have only shown
examples of replicating human faces, we predict that similar frame-
works will be used to design realistic full-body characters.
Acknowledgements
We would like to thank the anonymous reviewers for their com-
ments, Brian Tye, Frank Mezzatesta, and Walt Disney Imagineering
for their support, and Lee Romaire for figure finishing. This work
was supported in part by the NCCR Co-Me of the Swiss NSF.
References
BECKER, M., AND TESCHNER, M. 2007. Robust and efficient
estimation of elasticity parameters using the linear finite element
method. In SimVis, 15–28.
BEELER, T., BICKEL, B., BEARDSLEY, P., SUMNER, B., AND
GROSS, M. 2010. High-quality single-shot capture of facial
geometry. ACM Trans. Graph. 29, 4 (July), 40:1–40:9.
BEELER, T., HAHN, F., BRADLEY, D., BICKEL, B., BEARDS-
LEY, P., GOTSMAN, C., SUMNER, R. W., AND GROSS, M.
2011. High-quality passive facial performance capture using an-
chor frames. ACM Trans. Graph. 30, 4 (August), 75:1–75:10.
BICKEL, B., BOTSCH, M., ANGST, R., MATUSIK, W., OTADUY,
M., PFISTER, H., AND GROSS, M. 2007. Multi-scale capture
of facial geometry and motion. ACM Trans. Graph. 26, 3 (July),
33:1–33:10.
BICKEL, B., B¨ ACHER, M., OTADUY, M. A., MATUSIK, W.,
PFISTER, H., AND GROSS, M. 2009. Capture and modeling
of non-linear heterogeneous soft tissue. ACM Trans. Graph. 28,
3 (July), 89:1–89:9.
BICKEL, B., B¨ ACHER, M., OTADUY, M. A., LEE, H. R., PFIS-
TER, H., GROSS, M., AND MATUSIK, W. 2010. Design and
fabrication of materials with desired deformation behavior. ACM
Trans. Graph. 29, 4 (July), 63:1–63:10.
BONET, J., AND WOOD, R. D. 1997. Nonlinear Continuum Me-
chanics for Finite Element Analysis. Cambridge Uni. Press.
BRADLEY, D., BOUBEKEUR, T., AND HEIDRICH, W. 2008. Ac-
curate multi-view reconstruction using robust binocular stereo
and surface meshing. In Proc. CVPR.
BRADLEY, D., HEIDRICH, W., POPA, T., AND SHEFFER, A.
2010. High resolution passive facial performance capture. ACM
Trans. Graph. 29, 4 (July), 41:1–41:10.
BUCUR, D., AND BUTTAZZO, G. 2005. Variational Methods in
Shape Optimization Problems. Birkhuser Mathematics.
CHENTANEZ, N., ALTEROVITZ, R., RITCHIE, D., CHO, L.,
HAUSER, K. K., GOLDBERG, K., SHEWCHUK, J. R., AND
O’BRIEN, J. F. 2009. Interactive simulation of surgical needle
insertion and steering. ACM Trans. Graph. 28, 3 (July), 88:1–
88:10.
CLARBERG, P., JAROSZ, W., AKENINE-M¨ OLLER, T., AND
JENSEN, H. W. 2005. Wavelet importance sampling: efficiently
evaluating products of complex functions. ACM Trans. Graph.
24, 3 (Aug.), 1166–1175.
DONG, Y., WANG, J., PELLACINI, F., TONG, X., AND GUO, B.
2010. Fabricating spatially-varying subsurface scattering. ACM
Trans. Graph. 29 (July), 62:1–62:10.
GOURRET, J.-P., THALMANN, N. M., AND THALMANN, D.
1989. Simulation of object and human skin deformations in a
grasping task. In Comp. Graph. (Proc. SIGGRAPH), 21–30.
HARA, F., AKAZAWA, H., AND KOBAYASHI, H. 2001. Realistic
facial expressions by sma driven face robot. In Proc. of IEEE
International Workshop on Robot and Human Interactive Com-
munication, 504 –511.
HAˇ SAN, M., FUCHS, M., MATUSIK, W., PFISTER, H., AND
RUSINKIEWICZ, S. 2010. Physical reproduction of materi-
als with specified subsurface scattering. ACM Trans. Graph. 29
(July), 61:1–61:10.
IRVING, G., TERAN, J., AND FEDKIW, R. 2004. Invertible finite
elements for robust simulation of large deformation. In 2004
ACM SIGGRAPH / Eurographics SCA, 131–140.
KAUER, M., VUSKOVIC, V., DUAL, J., SZEKELY, G., AND BA-
JKA, M. 2002. Inverse finite element characterization of soft
tissues. Medical Image Analysis 6, 3, 257–287.
(a) (b) (c) (d) (e)
Figure 8: Illustrating the volumetric shape optimization in 1D. Given a volumetric object and a target surface (indicated in orange) as shown
in column (a), our optimization process computes the volumetric shape (b) such that the surface matches the deformation of the target under
actuation (c). Columns (d) and (e) show the undeformed and deformed state of the corresponding fabricated objects against a 4mm grid.
Figure 9: Side-by-side comparison. From left to right: Photographs of captured facial expressions; Renderings of 3D reconstructions; error
comparison of 3D reconstructions to the fabricated skin (blue 0mm, red 4mm); 3D simulation of the animatronic head for the corresponding
pose; error comparison of our simulation to the fabricated skin (blue 0mm, red 4mm); expressions generated with the optimized fabricated
skin actuated by an animatronic device.
(a) (c) (e) (g)
(b) (d) (f) (h)
Figure 10: Forehead Optimization: (a) Rest shape. (b) Target geometry. (b,c) Rendering of the inner side of the unoptimized and optimized
forehead with color-coded thickness variation. Comparison of the unoptimized and optimized skin: Error plot of the (e) unoptimized and (f)
optimized skin (blue 0mm, red 4mm). Although the variations are subtle, a comparison of the fabricated unoptimized (g) and optimized (h)
skin shows visible differences.
KHAREVYCH, L., MULLEN, P., OWHADI, H., AND DESBRUN,
M. 2009. Numerical coarsening of inhomogeneous elastic ma-
terials. ACM Trans. Graph. 28 (July), 51:1–51:8.
KOCH, R. M., GROSS, M. H., CARLS, F. R., VON B¨ UREN, D. F.,
FANKHAUSER, G., AND PARISH, Y. I. H. 1996. Simulating
facial surgery using finite element models. In Proc. of Comp.
graph. and int. tech., ACM, SIGGRAPH ’96, 421–428.
LEE, S.-H., AND TERZOPOULOS, D. 2006. Heads up!: biome-
chanical modeling and neuromuscular control of the neck. In
ACM SIGGRAPH 2006 Papers, ACM, 1188–1198.
LEE, S.-H., SIFAKIS, E., AND TERZOPOULOS, D. 2009. Com-
prehensive biomechanical modeling and simulation of the upper
body. ACM Trans. Graph. 28 (September), 99:1–99:17.
LEVIN, D. 1998. The approximation power of moving least-
squares. Math. Comput. 67 (October), 1517–1531.
LI, X.-Y., SHEN, C.-H., HUANG, S.-S., JU, T., AND HU, S.-M.
2010. Popup: automatic paper architectures from 3d models.
ACM Trans. Graph. 29 (July), 111:1–111:9.
MORI, Y., AND IGARASHI, T. 2007. Plushie: an interactive design
system for plush toys. In ACM SIGGRAPH 2007 papers, ACM,
SIGGRAPH ’07.
NESME, M., KRY, P. G., JE ˇ R ´ ABKOV´ A, L., AND FAURE, F. 2009.
Preserving topology and elasticity for embedded deformable
models. In ACM SIGGRAPH 2009 papers, ACM, 52:1–52:9.
NISHIO, S., ISHIGURO, H., AND HAGITA, N. 2007. Humanoid
Robots: NewDevelopments. I-Tech, ch. Geminoid: Teleoperated
Android of an Existing Person.
NOCEDAL, J., AND WRIGHT, S. J. 2000. Numerical Optimization.
Springer, August.
OH, J.-H., HANSON, D., KIM, W.-S., HAN, I. Y., KIM, J.-Y.,
AND PARK, I.-W. 2006. Design of android type humanoid robot
albert hubo. In Proc. of IEEE/RSJ Int. Conference on Intelligent
Robots and Systems, 1428–1433.
PAI, D. K., DOEL, K. V. D., JAMES, D. L., LANG, J., LLOYD,
J. E., RICHMOND, J. L., AND YAU, S. H. 2001. Scanning
physical interaction behavior of 3d objects. In Proc. of Comp.
graph. and int. tech., ACM, SIGGRAPH ’01, 87–96.
SCHENK, O., AND G¨ ARTNER, K. 2006. On fast factorization piv-
oting methods for symmetric indefinite systems. In Elec. Trans.
Numer., no. 23, 158–179.
SIFAKIS, E., NEVEROV, I., AND FEDKIW, R. 2005. Automatic
determination of facial muscle activations from sparse motion
capture marker data. In ACM SIGGRAPH 2005, 417–425.
SUEDA, S., KAUFMAN, A., AND PAI, D. K. 2008. Musculoten-
don simulation for hand animation. ACM Trans. Graph. (Proc.
SIGGRAPH) 27, 3.
TERAN, J., SIFAKIS, E., BLEMKER, S. S., NG-THOW-HING, V.,
LAU, C., AND FEDKIW, R. 2005. Creating and simulating
skeletal muscle from the visible human data set. IEEE Trans. on
Vis. and Comp. Graph. 11, 3, 317–328.
TERZOPOULOS, D., AND WATERS, K. 1993. Analysis and syn-
thesis of facial image sequences using physical and anatomical
models. IEEE Trans. Pattern Anal. Mach. Intell. 15 (June), 569–
579.
TERZOPOULOS, D., PLATT, J., BARR, A., AND FLEISCHER, K.
1987. Elastically deformable models. In Proc. Comp. graph.
and int. tech., ACM, SIGGRAPH ’87, 205–214.
THOUTIREDDY, P., AND ORTIZ, M. 2004. Avariational r-adaption
and shape-optimization method for finite-deformation elasticity.
Int. J. Numer. Meth. Engng. 61, 1–21.
VAN GELDER, A. 1998. Approximate simulation of elastic mem-
branes by triangulated spring meshes. J. Graph. Tools 3, 2, 21–
42.
WANG, Y., HUANG, X., LEE, C.-S., ZHANG, S., LI, Z., SAMA-
RAS, D., METAXAS, D., ELGAMMAL, A., AND HUANG, P.
2004. High resolution acquisition, learning and transfer of dyan-
mic 3-d facial expressions. In Comp. Graph. Forum, 677–686.
WEYRICH, T., PEERS, P., MATUSIK, W., AND RUSINKIEWICZ,
S. 2009. Fabricating microgeometry for custom surface re-
flectance. ACM Trans. Graph. 28 (July), 32:1–32:6.
ZHANG, L., SNAVELY, N., CURLESS, B., AND SEITZ, S. M.
2004. Spacetime faces: high resolution capture for modeling
and animation. ACM Transactions on Graphics 23, 3 (Aug.),
548–558.

In the context of computer graphics. ?. we use a dedicated parametrization approach combined with a moving least squares interpolation technique that allows us to modify the shape only in its thickness direction. has been carried out by Intelligent Research Laboratory at Osaka University. In contrast to our approach. we fabricate synthetic skins with uniform material properties and optimize their shape. ?. which are adequately modeled as elastic materials with isotropic and homogeneous properties. design. the design of reflectors for luminaires or the optimization of material interfaces. [?] was one of the first group to build an anthropomorphic head to study human-robot interaction. Hara et al. [?] describe a comprehensive system for capturing interactions with real-world objects. fabrication-oriented design is a recent trend that is seeing increasing attention in computer graphics. Robotics & Animatronics. There exists a multitude of methods for capturing facial expressions. nonlinear viscoelastic soft tissue [?]. the work by Teran et al. including deformation response. scared. • A comprehensive analysis and experimental validation of the process and its individual steps. ?]. surface roughness and contact sounds. There exists a plethora of dedicated bio-mechanical models for different parts of the human body such as the face [?. The head was able to reproduce six different expressions such as surprised. fabrication and validation. targeting the reproduction of geometric features of papercraft [?] or plush toys [?]. Measuring and Fitting Material Parameters. Such problems are typically cast as constrained optimization problems and solved numerically with finite elements (see. 4). The group has built a number of robotic characters with increasing capabilities. 3 Overview Our approach to physical face cloning comprises modeling. including data acquisition.g. ?]. Several methods have been presented in this context. as in our case. disgusted. . But since it is generally difficult to control the adaptation of the undeformed configuration. As a natural step to follow acquisition and data-driven simulation. In addition. Our approach for determining parameters of different silicone samples goes along similar lines: we use a vision-based acquisition system to capture dense surface displacement fields and determine optimal material parameters by minimizing the displacement error of a corresponding finite element approximation. including traditional marker-based motion capture [?. Android research. The individual steps of our processing pipeline are illustrated in Fig. We optimize the shape of the synthetic skin by minimizing an elastic energy with respect to rest state positions. angry. structured light systems [?. and material sciences. Similarly. The robot’s head resembles Albert Einstein and its underlying electro-mechanical actuation has 35 degrees of freedom. We start by capturing facial expressions of a human subject using an optical performance capture system that offers high-resolution 3D reconstructions including pores and wrinkles. Simulation. Examples of such shape optimization problems include the computation of aerodynamic profiles. ?. on the computational side. the hand [?. Their recent humanoid robot Geminoid HI-1 [?] as well as it’s successor Geminoid-DK have the realistic appearance of an adult man. several methods have been described for fitting measured data to computational models. The face skin was made of silicone and it was driven by SMA (shape memory alloy) actuators. which is similar in spirit to methods for variational shape adaptation [?]. optimization. the textbook by Bucur and Buttazzo [?] for an introduction). the individual components of our system have diverse connections to previous research. we rely on a compressible neo-Hookean material model and. e. 2. and passive markerless approaches [?. simulation. it has become common to model elastically deformable materials using continuum mechanics and finite elements. employ a finite element approach centered around linear tetrahedral elements. e. applied mathematics. Our acquisition system uses the recent method of Beeler et al. However. Physical face cloning is a complex task that intersects with numerous research areas from computer graphics. Introduced to the graphics community by Terzopoulos et al [?]. 2 Related Work Engineering applications often require the shape of a structure to be optimal with respect to given conditions. the design process for all these robotic skins is mainly manual. synthesizing and transferring facial expressions are important research topics in computer graphics. or even the entire upper body [?]. Face Capture. from measurements of realworld objects. The majority of robots consist of only rigid mechanical structures. These robots also have silicone-based skin driven by actuators placed underneath. [?] and references therein). and the implications of changes to the shape have to be tested in a time consuming trial and error manner. This reference data can originate either from an accurate computational model [?. Shape variations are modeled by an artist. Our approach for fabricationoriented material design bears some similarities to this last reference. Shape Optimization. physical simulation. a large body of literature is concerned with modeling of biological soft tissue (see. Although we are not aware of existing work to address this problem in its entirety.g. determine spatially varying material properties for fixed geometry. but rather take an output-oriented approach: we fabricate synthetic skin using silicone rubbers.. The skin is made of a sponge-like elastomer which is very soft and requires very little force to deform. ?]. Owing to the expressive power of the human face and our ability to infer a burst of information from the slightest motion. or nonlinear heterogeneous materials [?]. as well as robust temporal correspondence (Sec. KAIST and Hanson Robotics have developed a robot called Albert HUBO [?]. [?] to capture short performance sequences with high-resolution geometry and dense temporal correspondences. We do not strive to reproduce the mechanical properties of biological tissue. ?]. ?]. ?]. Pai et al. general appearance properties [?. ?. including linear-elastic materials [?]. ?] or. but there is also a crucial difference: whereas Bickel et al. • A complete process for automating the physical reproduction of a human face on an animatronics device. This data provides information about the deformation behavior of the subject’s skin.• A method to optimize for synthetic skin geometry and actuation parameters that provide the best match to a collection of given target expressions. and also the elastic properties of deformable media [?]. robots that are covered with soft-tissue materials such as rubbers or silicones have been built as well. various organs [?]. Closely related to the choice of a constitutive law is the problem of finding a set of parameters that best approximates a given reference model. which involves studying human-computer interaction.. sad and happy. More specifically. Fabrication-oriented Design. capturing. and fabrication of synthetic soft tissue. the torso [?].

The vector p holds the physical parameters of the material model. evaluate the objective function O(pi ) and determine updated parameters pi+1 . The measurement proceeds by pulling on small samples with controlled forces and capturing the resulting deformation over time using stereo reconstruction [?]. Silicone rubber is an ideal base material for fabricating synthetic skin since it is easy to process and offers a vast range of stretch resistance. All these degrees of freedom are handled in one uniform optimization framework (Sec. we also determine the operational range of the animatronic head. compatible mesh sequence with explicit temporal correspondence. Our approach is to capture a short performance of the target subject. Hence. we first measure the force-deformation behavior of a collection of silicone samples with different amounts of plasticizer. let ϕ : Ω → IR3 denote the mapping that transforms material points from the undeformed to the deformed configuration as We rely on real-world data for modeling facial geometry and expressions as well as material behavior. Measuring & Fitting Material Parameters. we also optimize the actuation parameters of the animatronic device to find the parameters that best resemble the individual target expressions. we numerically fit our computational model to experimentally acquired forcedisplacement samples of materials with different stiffnesses. The following paragraphs briefly describe our techniques for data acquisition and integration. we optimize the material parameters of our finite element solver in order to best approximate the measured stress-strain behavior. The result is a detailed. . To this end. Further. Considering the operating range of our animatronics device. we compare the stress-strain curves obtained from a standard testing procedure (ISO 37) to those obtained from a virtual counterpart of the same experiment. we chose 8 expressive poses from the sequence for optimizing the skin parameters. Since we have to account for large rotations. With a set of applied loads and resulting surface displacements determined. In each iteration. we have to adjust the silicone composition such that the net stiffness of the synthetic skin complies to these constraints. stretching. +50%]. To accomplish this task. we are able to match the real material behavior quite closely for deformations in the range of [−20%. 9). Material Sample Material Parameters Actuation Parameters Final Skin Acquisition Optimization Fabrication Figure 2: Physical face cloning pipeline. Face Scanning. [?]. the goal is now to design synthetic soft tissue that matches the example shapes as closely as possible. which imposes an upper bound on the net stiffness of the synthetic skin. Here we only give a brief overview and refer the reader to the textbook by Bonet and Wood [?] for a comprehensive exposition. leveraging the high resolution geometry acquisition method of Beeler et al. While the outer surface of the skin in the undeformed configuration is given by a 3D scan of the neutral target pose. 5 Physical Simulation of Synthetic Skin 4 Measurements and Acquisition As a central part of the face cloning pipeline. which allows us to sample a mapping of actuation parameters pact to 3D location and orientation of each link in space. We then compare predicted simulation results and real-world behavior for different deformation scenarios (Sec. we need to have an accurate computational model for simulating deformations of a synthetic skin. 8).Face Meshes f(pi)= Animatronic Head Skin Attachement Paths Material Thickness 3D Printed Mold When cloning a human face. 6). Our base material for the synthetic skin is silicone. and compression. In order to determine material parameters for the skin simulator. The molds are created using a rapid prototyping 3D printer. 4. As can be seen from Fig. which consists of a set of skin attachment links actuated by motors. which offers a wide range of stiffness that can be controlled by adjusting the concentration of plasticizer in the compound. Given an initial guess p0 we iteratively compute updated parameters in a simulated annealing process. The forward simulation allows accurate prediction of the deformed shape of a given synthetic skin and the resulting forces induced by the moving skin attachment links. Our approach is to model skin as a hyperelastic isotropic solid. this provides us with sufficiently large safety margins. matching the shape of the face and how the skin deforms under different expressions is one of the main challenges. Let X and x denote smooth functions describing the position of the skin in its undeformed and deformed state. we start by 2 defining an objective function O(p) = x i | (˜ i − xi (p)) | that measures the difference between simulated surface positions xi (p) ˜ and their closest corresponding point of the captured surface xi . We then perform numerical optimization in order to determine the parameters of our computational model that best match the experimental data. We must reconstruct various 3D expressions of the subject to be cloned. We attach to each moving link a marker. respectively. In our case. 5). Interpolating these samples yields a continuous function m(pact ) that describes the path of the skin attachment links (Sec. we resort to finite-deformation continuum mechanics. 5). Furthermore. The animatronics device can only exert a limited force through its actuators. We model and simulate the deformation behavior using a non-linear finite element method in combination with a neo-Hookean material (Sec. This process is repeated until the objective function signals convergence. we compute the finite element solution x(pi ) with the current parameters. in order to seed the deformable skin optimization. In order to design a synthetic skin that best matches a desired target. 7). which can be identified with Young’s modulus and Poisson’s ratio (see Sec. As a validation for this numerical-experimental material fitting process. Given the acquired example shapes of the human face and the description of the animatronic head. we propose to vary the inner surface (thickness) of the skin to achieve spatially varying stiffness and deformation behavior. we introduce a novel optimization process. We validate our simulation and optimization approach by fabricating various silicone objects using injection molding (Sec. allowing us to analyze the deformation of a face under various expressions. down to the level of individual wrinkles.

Figure 4: Force-strain curve of real (black) and simulated (red) material. xT )T and 1 n X = (XT . Having settled upon this material model. i. we note that the deformation gradient is constant e within each element since Na are piece-wise linear functions. The discrete deformation gradient for each element e is then expressed as 4 Fe = F(xe . XT )T denote vectors containing all deformed and 1 n undeformed positions respectively. which are discussed next. x The neo-Hookean material offers only two parameters. (4) where Ωe is the element’s parameter domain and V e its undeformed volume. but for the range of deformations considered here. respectively ∂X C = FT F .e. we let Xe ∈ IR12 and xe ∈ IR12 denote position vectors for a given element in its undeformed and deformed state.Figure 3: Illustration of thickness optimization. Summing up all elemental contributions. we discretize (1) in space using tetrahedral finite elements. The deformed configuration of the skin is then determined as the minimum of the total energy x = argmin (W (x. 2(3κ + µ) (2) We can now use this energy in a static equilibrium problem in order to compute the deformation of the skin in response to actuator placements. In particuar.2 Discretization The optimization process described in the next section requires the minimization of the elastic energy subject to boundary conditions. . which are related to the familiar Young’s modulus E and Poisson’s ratio ν as E= 9µκ . From now on. tr(Ce ) and J e . Mooney-Rivlin. (6) where Wext summarizes external contributions to the potential. 1 ≤ a ≤ n denote deformed positions of a tetrahedral mesh. Venant-Kirchhoff solid cannot be used in this setting [?]. Fig. . x = ϕ(X). . We proceed by approximating the configuration mapping as ϕ ≈ a xa Na where Na are piece-wise linear basis functions associated with the nodes of the mesh. . By solving (6). and Ogden material models are well suited for rubber-like substances such as silicone. 6 Optimization Recall that our goal is to fabricate a synthetic skin which. Requiring the gradient of the energy to vanish at equilibrium leads to a set of nonlinear equations. which translate into a set of position constraints. . 5. matches the target expressions as closely as possible. follow directly from (3). e. which is described in a rotationand frame-invariant manner through the tensor C. we determine the model parameters µ and κ for a given material sample using the numerical-experimental technique described in the previous section. respectively. Let xa ∈ IR3 .1 Material Model where we emphasize the fact that for each element Fe is a function of its position in both undeformed and deformed configuration. The deformation of the skin at each point can be characterized by the deformation gradient F or. which are solved in the standard way using Newton iterations with incremental loading and line search for convergence control. alternatively. X) + Wext (x)) . . Xe ) = Ωe Animating expressions on a virtual face generally induces both finite stretching and compression of the skin. when deformed by the underlying electromechanical base. 4 compares the force-strain curve for a measured and simulated material. the discrete elastic energy of a deformed element W e can be expressed as W e (xe . We will use the superscript e to identify quantities pertaining to a given tetrahedral element. X) = W e (xe . let x = (xT . 5. In order to achieve this requirement. (a) Target surface (b) undeformed skin patch (c) unoptimized skin patch deformed by vertical load (d) deformed skin patch with optimized thickness. we found this model to be sufficiently accurate for obtaining good correspondence with the measured data. Xe ) = a=1 xe a e ∂Na ∂X T . we obtain the total energy for a deformed configuration as W (x. Furthermore. the nonlinear neo-Hookean. κ are shear and bulk moduli. This process is subject to a number of . the strain energy density of the compressible neo-Hookean material is given as Ψ(C) = κ µ −2 (J 3 tr(C) − 3) − (J − 1)2 . 2 2 (1) Ψ(Fe ) = Ψ(Fe )V e (Xe ) . respectively 3κ + µ ν= 3κ − 2µ . All other discrete quantities required for computing the elastic energy of a deformed element.g. (3) For a hyperelastic material. To this end. Being the simplest representative of this class. Xe ) . due to gravity. the energy of the solid depends only on its current state of deformation. we optimize the shape of the skin as well as the actuation parameters of the base. . we can simulate skin deformation in response to arbitrary actuator placements. We denote the corresponding energy density function by Ψ = Ψ(C). (5) e where J = det F and µ. While the simple St. The relation between Ψ and C is described by material models. With this relation established. . the right Cauchy-Green tensor C which are defined as F= ∂ϕ .

X(p)) + Wext (x(ˆ. we embed a set of points in the simulation domain and deform them according to the mapping ϕ. p)). To this end. Furthermore. The undeformed geometry of the skin can now be described as X = r(u. p) = W (x(ˆ. we can now proceed to its optimization. We start by describing the general optimization framework and then how we apply it to the individual parameter sets. The matching energy measures how well a configuration x is in agreement with what we want the object to look like. p) denote the total energy of our physical system. we change the thickness of the skin by modifying its shape on the inside – we are free to do so everywhere except at the attachment points and in their immediate vicinity. In each Newton step. Second. We additionally regularize the parameters using a regularization energy Ereg (p). In our implementation. The parameters pthk determine the thickness distribution in the undeformed configuration without directly affecting the deformed positions. Having defined the thickness in this way. p) = x ˆ 1 ∂W γ 2 ∂ˆ x 2 and minimize (10) to find the parameter values for the thickness distribution that best match the given pose. Ematch (x) = 1 ||Sx − q||2 2 (11) where the matrix S contains the weights for the computation of the deformed points from x. we use a sparse direct solver [?] to solve the linear system Numerical Optimization. and we can write ˆ x W (ˆ. In the most general case. leading to the necessary condition of a vanishing residual force ˆ ∂W ∂ˆ x = 0. pthk ) = W (ˆ.p ˆ (9) The goal of the thickness optimization is to modify the local thickness of the skin in such way that when the mechanical actuators of the robot are set to the values corresponding to a particular pose. while Fig. thus we can write ˆ x W (ˆ.1 Generic Optimization Framework ˆ x Let the function W (ˆ. both the undeformed and deformed positions of our simulation mesh are allowed to depend on p. a condition we can easily detect by the direct solver not being able to factorize H. the physical system will assume the equilibrium state ˆ x xeq (p) = argmin W (ˆ. p) match a desired tarx get configuration as closely as possible. this allows the skin to produce bulges and wrinkles where they did not appear before. 6 shows some shape variations that can be . To this end. We employ line search to make sure the energy decreases in each Newton step. A roughly area-preserving map is desirable for obtaining unbiased sample distributions. this information is not explicitly available from the 3D geometry such that we have to infer it first. 6. All other values of x are either constant (constrained) or computable from p. ensuring with a sufficiently high penalty γ that the resulting x is close to a physiˆ cal solution with vanishing residual forces. xeq . including internal deformation energies and external contributions to the potential. X(pthk )) + Wext (ˆ) x x (13) The goal of the optimization will be to find the optimal parameter values p such that the positions x(ˆeq (p).constraints. v) from a sparse set of height samples αi at positions (ui . x x (7) H ∆ˆ x = −f ∆p (12) for the increments ∆ˆ and ∆p. the motion of the actuators is physically restricted. First. we use a subset of points from high-resolution 3D scans (Sec. ˆ with the following objective function: E(ˆ. Fig. We cannot change the outside surface of the skin. we compute the second derivatives of the first term in (10) only approximately. ignoring the third order derivatives of W that would result from applying the chain rule. It can happen that the matrix H becomes indefinite. the actuator settings result in hard positional constraints that can directly be applied to the corresponding deformed positions. Practically. In the physical simulation. x contains a subset of the deformed ˆ positions x of the physical simulation. Although the skin clearly exhibits a thickness direction. Due to our choice of basis functions. the resulting deformation of the skin matches the pose’s target positions q as closely as possible. For fixed parameter values p. we have to expose as many degrees of freedom as possible to the optimization process. We solve this as a minimization problem with respect to x and p. v). 4). This is measured in terms of a matching energy Ematch (x). but this is not a critical aspect in this context. We also cannot modify the attachment points of the actuators. p is a vector of generic parameters for which we want to optimize. In order to achieve the best results given these constraints. We construct a smooth warping field w(u. we optimize the actuation parameters of the animatronic device for every target expression. p) ˆ x ˆ 6. We minimize (10) using a Newton method. v)+hn(u. We then compute the matching energy as Matching Energy. 5 illustrates the concepts of parametrization and MLS interpolation. where u and v are surface coordinates and the thickness parameter h runs along the surface’s normal direction n. For this we need to compute its first and second derivatives with respect to x and p. Instead of working directly on the undeformed positions. Parameterizing the Undeformed State. + Ematch (x(ˆ. in which case we proceed as suggested in Nocedal and Wright [?] by adding a multiple of the identity to H.p ˆ The first term penalizes the violation of condition (9). p). we define a warping heightfield on the surface in order to smoothly deform space along its normal direction. While these computations are trivial ˆ for our particular choices of matching and regularization energies. we optimize with respect to two sets of parameters. the deformed position of a point can be computed as a linear combination of four nodal positions. vi )T using moving least squares (MLS) interpolation [?]. where f is the vector of first derivax tives of E and H contains its second derivatives. We start by computing a low-pass filtered version of the skin’s outer surface and construct a parameterization r : IR2 → IR3 using standard uv-mapping software. p)) + Ereg (p) (10) x x.2 Thickness Optimization (8) minimizing the total energy of the system. the squared distance between the deformed points and their desired target positions q.

va . Fabrication Constraints. We have ˆ ˆ To actuate the silicone skin we use a proprietary animatronic head developed by Walt Disney Imagineering. ui . thus the matrix ∂X/∂pthk is constant and can be precomputed. v) (1. Additionally. 6. Once the optimal αi are determined. v) amounts to fitting an affine transformation that minimizes the weighted least squares error θ(||(u. . In the objective function (10) we use ˆ x W (ˆ. vi ) ||)||a(u. Note that the MLS interpolation results in a linear mapping between undeformed positions X and parameters pthk . In this setting. . . there are several rigid attachment . vi ) − αi || .pthk ˆ i Ematch (x(ˆi . (19) The mapping from unconstrained nodal positions x and parameters pact to deformed positions x is ˆ xi = xi ˆ Ma (pact )Xi if node i unconstrained (20) if node i constrained by actuator a. this energy will tend to infinity as any element’s volume approaches zero. Ereg (pthk ) = γundefreg W (X(pthk ). or falling below a user-defined minimum thickness. where θ(x) is a weighting kernel and a(u. we use the regularization energy Ereg to keep the actuator parameter values within the range of physically feasible values. pact ) = W (x(ˆ. (a) Input geometry (b)–(d) examples of warped geometry. i m matching functions Ematch and the objective function can be written as m obtained for a simple one-dimensional example. Similar to the thickness optimization. evaluate the MLS warping field to obtain wa = w(ua . abusing X(pthk ) as the deformed configuration and the initial undeformed positions X(1) as the undeformed configuration: Regularization. we found it most practical to reuse the deformation energy function W . x x Parameterization. Figure 6: Illustration of MLS interpolation using 7 sample points. va ) and finally compute the warped positions as Xa = r(ua . effectively preventing element inversions in the undeformed configuration. (14) i T T T 2 E(ˆ. and use pthk = 1 for the MLS parameters. pact )). X(1)) (15) For the sake of simplicity. pact ). v) ∈ IR3 are the local coefficients of the interpolation function. va ). Evaluating the MLS warping field at an arbitrary parameter point (u. the given target positions qi should be matched as closely as possible. We need the undeformed mesh with nodal positions X(pthk ) to be a valid tetrahedral mesh where every element has positive volume. va ) + wa ha n(ua .3 Actuation Parameter Optimization The goal of the actuation parameter optimization is to find the best values for controlling the mechanical actuators such that the resulting deformed skin matches a given target pose as closely as possible. The latter are determined by solving the normal equations for (14). we adaptively place the locations of the height samples in uv space based on a user-provided density map [?]. the final rest configuration is obtained in the following way: for each point Xa of the original mesh we first retrieve its parameter values (ua . which can be done efficiently using precomputations. Regularization. the parameters pact of the optimization define the positional constraints to apply to the nodal positions x. Furthermore. 7 Animatronic Head We want to optimize the undeformed configuration X(pthk ) such that m > 1 poses can be reproduced as closely as possible: for a set of actuator settings ai . Right: MLS warping field shown as surface with isoparametric lines for illustration. X) + Wext (x(ˆ. v) − (ui . . To control the resolution of the warping heightfield. we add a quadratic energy term to Ereg that penalizes MLS weights exceeding a value of one. This guarantees a fabricatable skin that does not intersect the underlying mechanical structure. The length of the red arrows illustrates the range of potential motion. Finally. xm and pthk at once. we gather the parameters αi of all sample points into pthk and compute optimal height values by minimizing (10). For any positive penalty value γundefreg . Multiple Pose Optimization.Figure 5: Left: parametrization of the geometry and placement of MLS samples. pthk )) x i=1 (17) (18) +Ereg (pthk ). We initialize the optimization with values xi computed from a perˆ pose forward simulation with incremental loading. ha ). pthk ) = x + i=1 m ˆ ∂W 1 γ 2 ∂ˆ x 2 (16) xi . While there are different ways to formulate and enforce this constraint. we linearize the transformation matrix Ma (pact ) of actuator a around the current parameter values pact in each step of the optimization. In this case our optimization is slightly modified and we optimize for all deformed positions x1 . It is driven by electric motors and features a = 13 parameters pact ∈ IRa to control the movement of the attachment links indicated in red in the figure on the left.

To obtain the specific animation parameters for a given facial expression. Silicone belongs to the class of elastically deformable polymers with a high yield strength. Our optimization process computes the volumetric shape of soft-tissue material under actuation such that the surface matches the deformation of given target surfaces. As matching criteria we select small surface regions close to the attachment links. We started by acquiring more than 100 frames of an expressive performance of our subject.35 mm. all experiments. Figure finishing included industry-standard techniques for adding hair and facial stubbles and painting a texture by a professional makeup artist. The difference between the scans of the real face and the simulation is larger. the target is not well approximated. To determine the operating range and control parameters of the head we attach a marker to each moving link and track its position under a number of sample locations and orientations. For our experiments.g. In our experiment. Tracking is performed using the same capture system as used for face scanning (Sec. the color can be adjusted. To better represent important facial deformation features on the forehead. it has a liquid consistency and can be injected into the mold. we use liquid injection molding to fabricate the physical skin as shown in the accompanying video. we used a subset of 8 expressive poses for thickness optimization. so in theory our optimization algorithm can be used with any animatronics device. and then showcasing our method with an example of real physical face cloning for an animatronic figure. This shows that our simulation is a good model for the real synthetic face. 9. As shown in Fig.links that constrain the movement of the skin. We validated our pipeline for measuring. 6. In those regions. 9 Results In this section we present our results.3. The mapping m(pact ) : IRa → IRl·6 from animation parameters pact to 3D locations and orientations of attachment points is then specified by quadratically interpolating the samples. we apply the same warping transformation to the captured geometry before doing comparisons (such as the error plots in Fig. modeling. we validate the simulation by comparing simulated deformations of our silicone skin with 3D scanned poses of the actual animatronic figure in Fig. The measurements are acquired by running an uniaxial stretch and compression test on a rectangular (55 × 5 × 2 mm — stretch) and cylindrical (50 mm diameter. 10) with our results. representing the limitations in the range of motion of the robot head and the range of deformation of the attached skin. We then aligned the neutral pose to a scan of the animatronic head. We showcase our complete pipeline by physically cloning a human face for an animatronic figure. 7. For all other regions we use our optimization process to achieve the desired deformation behavior by varying the skin thickness. After having optimized the ideal skin geometry. 9 shows our figure in action and compares it to the human face and the simulation under various facial expressions. 4). we additionally use a high-resolution mesh of ≈ 157k elements to optimize this area. We also scanned these poses of the animatronic figure and provide error visualizations showing the distance between the simulated surface and the scanned surface. our algorithm was able to exactly reproduce the spatially-varying thickness of the ground truth objects that were used to generate the inputs. In Validation Optimization. and compared the deformation of the predicted optimized shape to the real object. using material with Young’s modulus of E = 78kPa and Poisson’s ratio of ν = 0. we slightly warp the acquired 3D geometry of the human face to prevent intersections. with less than 4mm separating the surfaces everywhere. We start by generating ground truth data by randomly varying the thickness of a uniform block using the moving least squares warp as described in Sec. and the fabrication process is easy and safe. Fig. 4 shows a comparison of the force-strain curve of a simulated material to a measured sample. As can be seen from the error plots. Approximately seven days of room-temperature curing is required until it is ready to be used. The gap between the surface of the neutral pose and the underlying animatronics defines the initial unoptimized skin. Validation Simulation. We then used these target surfaces and the undeformed block as input into our optimization — see e. 12 mm height — compression) object. and simulating silicone materials.2. Our mold consists of a core that resembles the inner surface of the skin and six outer parts. our optimization process computes the thickness variation (c) such that the surface matches the deformation of the target under actuation (d). we also fabricated a subset of these examples. Because the animatronic head has very pronounced cheek bones. From the forward simulation of these objects we extract target surfaces. These are then used as hard boundary constraints in our forward simulation. starting with validation of our pipeline and optimization process. Fig. we solve the inverse problem as described in Sec. showing an average error of less than 1. Note that multiple outer parts are required so that the skin can be removed without destroying the mold. providing an intuition of which expressions are inside or at the border of the gamut of reproducible poses. Fig. For these 95mm long bars. we used the two-part silicone GI-245 by Silicones Inc. the agreement between the shape predicted by our simulation and the actual observed shape of the actuated skin is generally good. Consequently. Without optimization (e). We refer to the video for a visualization of the actuation level of each actuation parameter for a facial animation sequence. Fig.44mm (top) and 0. it is robust. After mixing it with the corresponding catalyst. The mold itself is fabricated using a rapid prototyping 3D printer.47 in its cured state. we also fabricated the optimized skin. 8 Fabrication Our choice to use silicone is based on the fact that the stiffness can be controlled accurately. We treat the underlying mechanics and controlling mechanism as a black box. Furthermore. which we represent as a tetrahedral mesh consisting of ≈ 63k elements. 6. the average errors between the deformed fabricated surface and the goal shape are 0. Comparing the human face to the cloned synthetic face.38mm (bottom). 8. Starting with a volumetric object (a) and a target surface (b). the large scale deformations and general expressions . Finally. We validated the optimization process in simulation. Face Cloning. 10 illustrates the unoptimized and optimized geometry and provides a side-by-side comparison of the deformation behavior. the surface thickness is constrained and cannot be changed because the skin is connected to the moving and static links. We then use our optimization process to obtain animation parameters for the entire captured face performance sequence. (a) (b) (c) (d) (e) Figure 7: Volumetric shape optimization.

4 (August). Cambridge Uni. B. B... P FISTER . AND BA JKA . N. D ONG . AND KOBAYASHI . R. M ATUSIK . M ATUSIK . Graph. C HENTANEZ . H. B ONET. AND F EDKIW. ACM Trans. T. Moreover. Limitations and Future Work. A.. K AUER . R. M. Press. e. R. Nonlinear Continuum Mechanics for Finite Element Analysis. W. M. 29 (July). Graph. AND G UO . H. 2005. B ICKEL . L.. (Proc. we currently only use a single soft tissue material to produce a synthetic skin. Finally. M. C HO . M.. W. 2010. This work was supported in part by the NCCR Co-Me of the Swiss NSF.. Wavelet importance sampling: efficiently evaluating products of complex functions. 1166–1175. H ARA . ACM Trans. Relaxing this constraint would likely give us more degrees of freedom... G OLDBERG . In Proc. M. D. In Comp. P. A KAZAWA . We believe that our computationally guided process for designing and fabricating animatronic characters will serve as a blueprint for how future robots with soft tissue should be built. G ROSS . 2007. Because this work is the first in the area. H EIDRICH . Graph.are captured well. 131–140. 2010. G OURRET. J. ACM Trans.). 29 (July). B.. 2009. P OPA . R. M. S UMNER . 3 (July). 29. P FISTER .. B. H. W. B EARDS LEY. M.. B. JAROSZ . AND B UTTAZZO . AND T HALMANN . B EELER . Acknowledgements We would like to thank the anonymous reviewers for their comments. A. J.. CVPR.. Graph.. WANG . AND G ROSS . 10 Conclusion In this paper we have described a process for computationally guided design of an animatronic character composed of a soft-tissue material and an electromechanical base. ¨ B ICKEL .. H AUSER . ACM Trans. A. ACM Trans. P FISTER . P.. 3 (Aug. F. M. V.. 3. Our experimental validation shows that using this process provides a principled way to design soft-tissue animatronic characters. D. D. ACM Trans.. T ERAN . 26. T ONG . Design and fabrication of materials with desired deformation behavior. A LTEROVITZ . D. 257–287. and the differences show where improvements should be made in the physical system to enable the fitting methods to achieve a better match. ¨ B ICKEL ... High resolution passive facial performance capture. K. Accurate multi-view reconstruction using robust binocular stereo and surface meshing. High-quality single-shot capture of facial geometry. B ACHER . In Proc. ACM Trans. H AHN . 1989. References B ECKER . I RVING . and hair. 4 (July).. 61:1–61:10. B ACHER . M. P. Medical Image Analysis 6. B ICKEL ... W. in our current framework the attachment of the actuators is fixed to specific places underneath the character’s skin. B UCUR . D. We believe we can extend the range of possible deformations by using multi-layered materials similarly to Bickel et al. 2010.. 24. K. 3 (July). M. A KENINE -M OLLER . Capture and modeling of non-linear heterogeneous soft tissue. H. K. 89:1–89:9. B OTSCH .. H. 29. B ICKEL ... of IEEE International Workshop on Robot and Human Interactive Communication. 2010. . B OUBEKEUR .. OTADUY.. it is important to note that the motions of current animatronic characters are still limited as they are less expressive than most humans. AND T ESCHNER . Realistic facial expressions by sma driven face robot. Invertible finite elements for robust simulation of large deformation. In SimVis. 41:1–41:10. Multi-scale capture of facial geometry and motion. L EE . A NGST. Simulation of object and human skin deformations in a grasping task. V USKOVIC .. 29. Graph... G. B RADLEY. First. In 2004 ACM SIGGRAPH / Eurographics SCA. 33:1–33:10. 2009. S UMNER . Graph. 4 (July). 28. 4 (July). 2011. B RADLEY. B EARDSLEY.. J.. 2008. OTADUY. T HALMANN . AND M ATUSIK .. M ATUSIK . ACM Trans... Graph. N. [?]. AND RUSINKIEWICZ . it is important to develop retargeting algorithms that can map the expressions of a target human to the physical constraints of the electromechanical base and materials used. J. AND W OOD .. and Lee Romaire for figure finishing. 63:1–63:10. 2002. Robust and efficient estimation of elasticity parameters using the linear finite element method. 504 –511. In this paper.. W. B. 2004. T. T. 1997. Variational Methods in Shape Optimization Problems.. ACM Trans.-P. SIGGRAPH). B. M. Graph. X. D... AND O’B RIEN . M. H. Physical reproduction of materials with specified subsurface scattering. G. while in this work we have only shown examples of replicating human faces. D UAL . H. Inverse finite element characterization of soft tissues. H. High-quality passive facial performance capture using anchor frames. we predict that similar frameworks will be used to design realistic full-body characters..... Interactive simulation of surgical needle insertion and steering. F.... R. ACM Trans. OTADUY. D.. J.. Brian Tye. it has a number of limitations that should be addressed in future research. T. the rest shape of the soft tissue would not need to be the same as the rest shape of the electromechanical base. AND H EIDRICH .. F UCHS . M. 28.. and Walt Disney Imagineering for their support. 88:1– 88:10.. M. AND G ROSS ... AND G ROSS . B EELER . F. ˇ H A S AN . We compute a volumetric shape of the soft-tissue material and a set of actuation forces that can reproduce target deformations of an exterior character surface. Y.. 15–28. W. 21–30. M. subsurface scattering. W. R ITCHIE . Graph. The core requirement of this process is a physically based simulation framework that lets us accurately predict the behavior of the soft tissue when subject to external forces. Graph. G OTSMAN . S. W.. ¨ C LARBERG .. S ZEKELY. we have not addressed the appearance aspect of the synthetic skin tissue including reflectance. Frank Mezzatesta. F. 2001... R. B RADLEY. 62:1–62:10. As an example. J. Birkhuser Mathematics. 75:1–75:10. 30. M. Graph. S HEWCHUK . 2007.. J. C. AND G ROSS . 3 (July). 2005.g. AND J ENSEN . M. T. P ELLACINI . G. this process allows us to construct an animatronic head that approximates the shape and behavior of a real person whose expressions are acquired using a 3D scanning system.. Second. This is also a very interesting direction for future work and we envision that similar computationally guided processes could be developed to solve these problems. Fabricating spatially-varying subsurface scattering. 2010.. Thus. AND S HEFFER . M. R. W. 40:1–40:9. For further comparisons we refer to the accompanied video.. P FIS TER .

error comparison of our simulation to the fabricated skin (blue 0mm. Figure 9: Side-by-side comparison. red 4mm). Renderings of 3D reconstructions. expressions generated with the optimized fabricated skin actuated by an animatronic device. Columns (d) and (e) show the undeformed and deformed state of the corresponding fabricated objects against a 4mm grid. error comparison of 3D reconstructions to the fabricated skin (blue 0mm. .(a) (b) (c) (d) (e) Figure 8: Illustrating the volumetric shape optimization in 1D. our optimization process computes the volumetric shape (b) such that the surface matches the deformation of the target under actuation (c). red 4mm). From left to right: Photographs of captured facial expressions. Given a volumetric object and a target surface (indicated in orange) as shown in column (a). 3D simulation of the animatronic head for the corresponding pose.

T ERZOPOULOS . 2009. Springer. T HOUTIREDDY. . L. Numerical coarsening of inhomogeneous elastic materials.. P.. ACM Trans. D.. IEEE Trans.. VAN G ELDER . H. Conference on Intelligent Robots and Systems.. T ERZOPOULOS . S. J E R ABKOV A . K. C. K. Graph. 569– 579. 1998.. ¨ KOCH . L I . SIGGRAPH ’87. SIGGRAPH) 27. J.. H ANSON . W. L EVIN . red 4mm). K HAREVYCH . N ISHIO .-M. L. K. D.. High resolution acquisition. of Comp. Design of android type humanoid robot albert hubo. and int.. 1996. D. 677–686. H. R ICHMOND . D. E. 2004. S AMA RAS .-S. Simulating facial surgery using finite element models. 99:1–99:17. AND H UANG . K. Graph.. N G -T HOW-H ING . The approximation power of moving leastsquares. WANG . L AU . S. VON B UREN . M ULLEN . Z HANG . 67 (October).-H. Preserving topology and elasticity for embedded deformable models. L EE . E. S. M ATUSIK . N. H. no. In ACM SIGGRAPH 2006 Papers... R. L EE .. 2006. 2007.. K RY. AND G ARTNER . B LEMKER . J. 3. In Proc. tech. In ACM SIGGRAPH 2009 papers. D. 2005. M. Trans. Pattern Anal. Automatic determination of facial muscle activations from sparse motion capture marker data. 11.. H.. 51:1–51:8. AND T ERZOPOULOS . R. M ORI .-H. OWHADI .. L LOYD . O H . ACM Trans.. (b. Tools 3. 1517–1531. 29 (July). J U . Z. JAMES .. Musculotendon simulation for hand animation. K IM . D. 15 (June).. J. ACM. Intell. O. Comprehensive biomechanical modeling and simulation of the upper body. F. of IEEE/RSJ Int.... I SHIGURO . Numer. Fabricating microgeometry for custom surface reflectance. T. C..-Y. Meth. BARR . graph. 2009. I. E. ˇ´ ´ N ESME . 2005.. Y. ACM. J. M ETAXAS . T ERAN . L. AND FAURE . In ACM SIGGRAPH 2007 papers. 52:1–52:9. X. S.. In Proc. 205–214... Graph. AND O RTIZ . D . P. 158–179... 87–96. AND I GARASHI . and int. 3. AND F EDKIW. M. 2000. Elastically deformable models. 21– 42. L EE . Comparison of the unoptimized and optimized skin: Error plot of the (e) unoptimized and (f) optimized skin (blue 0mm. 1987. C. D.. S HEN . I. M..-S. P EERS . P.-H. ACM... C ARLS . (b) Target geometry. Graph.. Analysis and synthesis of facial image sequences using physical and anatomical models.. J.. 61. Graph. J. Comp..).. 2001. Geminoid: Teleoperated Android of an Existing Person. I. S... D. A variational r-adaption and shape-optimization method for finite-deformation elasticity.-Y. AND PAI . S IFAKIS . 3 (Aug. L I .. AND T ERZOPOULOS . graph. SIGGRAPH ’07. 2004. L. S IFAKIS . H UANG . S UEDA . Graph. and int. Plushie: an interactive design system for plush toys. graph. R. K AUFMAN . on Vis. S. Comput. L ANG . AND F EDKIW. 2009.. ACM Trans. and Comp. A.. Creating and simulating skeletal muscle from the visible human data set. R. 2006. S. Engng.. 417–425. V. In Comp.. S. W EYRICH . S NAVELY. I. Spacetime faces: high resolution capture for modeling and animation. 2010. 28 (September).. Heads up!: biomechanical modeling and neuromuscular control of the neck. August. 1993. ¨ S CHENK . AND H U . S.. M. W. (Proc. C URLESS . 2. a comparison of the fabricated unoptimized (g) and optimized (h) skin shows visible differences. AND H AGITA . I-Tech. Mach. D. In Elec. AND D ESBRUN .. X. Scanning physical interaction behavior of 3d objects.. IEEE Trans. S. A. G ROSS . AND YAU . 317–328. D. H.-S. 1188–1198. 1–21.. ACM Transactions on Graphics 23. S. FANKHAUSER . ACM Trans. T. K. A. learning and transfer of dyanmic 3-d facial expressions. Forum. ACM Trans..c) Rendering of the inner side of the unoptimized and optimized forehead with color-coded thickness variation.. tech. 28 (July).. AND S EITZ .. Math.. H AN .-H. 111:1–111:9. AND WATERS . Numerical Optimization. 32:1–32:6. 28 (July). 2004. M. 1998.(a) (c) (e) (g) (b) (d) (f) (h) Figure 10: Forehead Optimization: (a) Rest shape.. Int. On fast factorization pivoting methods for symmetric indefinite systems. Approximate simulation of elastic membranes by triangulated spring meshes. ACM. N. Y. D. T. G.. D. N OCEDAL .-W. AND RUSINKIEWICZ . F. Z HANG . 1428–1433. ACM. Graph.. Although the variations are subtle.. 421–428. 2009. P. In ACM SIGGRAPH 2005. K IM . S. AND PARISH . V. In Proc. N EVEROV. of Comp. SIGGRAPH ’96.. E LGAMMAL .. S. In Proc.. J. Numer. P LATT. AND F LEISCHER . J. Y. tech. K. Y. J. L.. ACM. 23. 2008. Humanoid Robots: New Developments.. E. AND W RIGHT. S IFAKIS . H UANG . 2006. Popup: automatic paper architectures from 3d models. 548–558. B. PAI . 2007. A. Graph. M. G.. ch. J. SIGGRAPH ’01. J. D OEL . AND PARK . P. F.