Notes on downloaded

slides:
The accompanying videos can be
downloaded from:
www.ChrisEvans3D.com/tutorials.htm
The videos were captured with various
software which sometimes seriously butchers
the framerate, please keep this in mind
We have added comments generally
summing up what we said about
each slide
The audio of our session in mp3
format can be purchased from ‘GDC
Radio’ (
http://store.cmpgame.com/product.php?id=1847&cat=
)
Christopher Evans
Senior Technical Artist, Crytek

Responsibilities:
R&D Artist
Artist tools
Getting characters in-game
Hanno Hagedorn
Senior Character Artist, Crytek

Responsibilities
Character Heads
Modeling
Texturing
Rigging
Facial Setup
Character Bodies (partially)
What is this stuff?
A cross section of current
character-related R&D at
Crytek
Mostly WIP, all real-time, real
assets
Presenting collaborative work
Artist-driven, or real world
data-driven tools
Intended Audience
Intermediate to Advanced –
Requires experience and
familiarity with the subject
Technical Artists (TA) and
artists with technical interests
Not too complicated
Simple, pragmatic approaches
Session Overview
Low Level
Deformation
How you drive deformations
How you deform geometry
Animation
High Level
Character Variation
Case Studies
Internal Tools
Asset Creation
Facial Editor
Setup
Animation
What is a
 Rejected internal concept art

What is a Character?
Person or creature?
 Rejected internal concept art

What is a Character?
Vehicle or machine with
moving parts?
What is a Character?

Any object or set of

objects whose
movement is derived
from a skeleton.
What is a Character?
People and creatures
Vehicles and machines
Moving character attachments
Guns, backpacks, heads, hands
Dynamic, jointed physics
simulations
Ropes, complex constraint setups
Pre-baked physics and broken
objects
Pre-baking cinematic destruction
VIDEO:
00:04
The Low Level:

Deformatio
n

George Bridgman, The Human Machine 1929

The Low Level:

Deformation
What drives the
deformation?
Motion?
Arbitrary inputs?
Shader-based?
How do you move and
deform geometry?
Is it efficient?
What kind of approach?
VIDEO: 00:45

VIDEO:
00:45
The Low Level:

Deformation
What drives the
deformation?
Motion?
Arbitrary inputs?
Shader-based? (procedural noise)
How do you move and
deform geometry?
Is it efficient?
What kind of approach?
Driving Deformation:

Pose Driven
Systems

George Bridgman, The Human Machine 1929

Driving deformation:
Pose-Driven Systems
You can drive many things with
skeletal poses:
Morph Targets
Joints
Maps
Outside our industry: Not a new
idea at all
Simple 3 Vector Driver:
Shape

VIDEO:
01:03
Simple Driver: Color and
Shape

VIDEO:
01:09
Driving deformation:
Pose-Driven Systems
Pre-baked
Baked to keys/shapes
Runtime
In engine
Pros/Cons:
Pre-Baked Pose-Drivers
Pros:
Setup is in the asset creation app
Implementations work well in most
animation-based games (joints)
Cons:
Non-dynamic - Does not work with
motion generated at runtime
Ragdoll physics
Physically-based motion
Other procedurally-generated motions
Driving deformation:
Pose-Driven Systems
Pre-baked
Baked to keys/shapes
Runtime
In engine
Driving deformation:
Runtime Pose-Drivers
Pros:
Works dynamically in any situation
Only limit is the number of poses you
define
Cons:
Difficult to set up
Need to mirror in the asset package
Constraints
Parameterization
Driving deformation:
Arbitrary-Drivers
Holy grail of driven systems
General Arbitrary Drivers
General input/output system
Houdini, Shadetree, etc…
The more general the better
Runtime expressions?

Not limited to deformation at

all!
VIDEO:
01:16
VIDEO:
02:11
VIDEO:
02:54
VIDEO:
03:19
The Low Level:

Deformation
What drives the
deformation?
Character motion? Skeletal motion?
Arbitrary inputs?
Shader-based? (procedural noise)
How do you move and
deform geometry?
Is it efficient?
Is it an Artist-centric approach?
Deformation:

Skinning

George Bridgman, The Human Machine 1929

Linear Skinning
Weight vertices in a linear fashion to n bones

Definitely nothing new

First use in a game (software)
Trespasser? (1997-1998)
First use in a game (hardware)
dx8 via vert shaders
dx7 rarity
Hardware ‘deceleration’
Linear Skinning,
or smooth skinning, or linear blend
skinning..

Pros
Industry Standard
Lots of available research
Cons
Poor volume preservation
The more a joint bends, the more it collapses
Difficult to get specific control
Can be more resource intensive
Our Implementation of:

Spherical Skinning
“Hardware Skinning with
Quaternions” (2004)
by Jim Hejl (EA) Game Programming
Gems 4
First game to utilize it?
Pros/Cons

Spherical Skinning
Pros
Much better volume preservation
Less resource usage
(Skinning position, tangent, and bi-
normal)
72 bones per draw call
~48 (Linear w/ matrices)
~72 (Linear w/ quats)
20 instructions
~40 instructions (Linear w/ matrices)
~60 instructions (Linear w/ quats)
VIDEO:
04:11
VIDEO:
04:45
Pros/Cons

Spherical Skinning
Cons
Needs support in the asset creation
package
Plugin, preview, etc…
Inherent issues with some weighting
configurations
N bones per vert
Non-consecutive or interleaved bones
Interleaved
Bones
Example: Character Studio
Biped
Pros/Cons

Spherical Skinning
Cons
Needs support in the asset creation
package
Plugin, preview, etc…
Inherent issues with some weighting
configurations
N bones per vert
Non-consecutive or interleaved bones
The volume preservation sometimes
corrects things we have come to rely on
linear skinning for
Interested in New
Skinning Technology?

GDC Session: Skinning with Dual

Quaternions

Tomorrow (Fri)
9am – 10am
Room 2009, West Hall
Deformation:

Morph Targets

George Bridgman, The Human Machine 1929

Deformation:
Morphs are Your
Friends
Also known as:
Blendshapes, shapes, vertex animation
Artistic approach – Artist
defines the look completely
Facial animation
Sculpt muscles, phonemes, visimes
Corrective shapes
Driven by arbitrary inputs
Baked simulations
Bending metal, ocean waves, soft tissue
Deformation:
Morphs are Your
Friends
Keep them light in engine
Sparse morphs
Only the verts that move
Corrective shapes can be *very* light
Only move the points that you have to
Sparse morphs on DX10 hardware
Sparse morphs running on dx10 hardware
Supports binding to more than 4 color
images
Store morphs in a texture
Blend between textures
There is a demo in the DX10 SDK
They also have wrinkle maps in this
example!
Driving Morphs

Pose-Driven morphs
Artist-Sculpted Pipeline
Define pose vectors
Artist sculpts the pose
Link sculpts to those pose vectors
Parameterize/blend between poses
Simulation-Sculpted
Artist creates deformation
Links deformation to some arbitrary
parameter in the engine
Driving Morphs
Our asset creation tool
Allows the artist to pose the rigged
character
We decided to use pre-defined pose vectors
Generates a mesh in that pose for him to
sculpt
Extracts the artists sculpt with the skin
deformation culled
Created a shape for that pose
VIDEO:
05:26
VIDEO:
06:43
Our In Engine Setup
Apply the shape locally before the
skinning
Assign the shape to the correct bone and
pose vector
This is the simplest scenario
Single pose vector
VIDEO:
06:49
This is a simple test,
there are many more
uses!
Corrective morph shapes
SAY GOODBYE TO
SKINNING ARTIFACTS!
Complex deformation
You don’t have to sculpt these
targets by hand!
The sky is the limit
VIDEO:
09:14
VIDEO:
10:35
Deformation:

Helper Joints

George Bridgman, The Human Machine 1929

Helper Joints
Jason Parks: Advanced
Deformations on Runtime
Characters – GDC 05/06
http://www.jason-parks.com/HelperJoints/
http://www.jason-parks.com/MuscleSystems/

Great but less artist-driven

Universal
Needs a lot of technical support
Non-intuitive
Examples
Same driving issues as with
morphs
Pre-baked vs. Runtime
VIDEO:
10:46
Example: Shark
Example:
Nanosuit
Example:
Nanosuit
Example:
Nanosuit

Helper Joints (12):

Deltoids (shoulders) (8 joints)
4 on each shoulder
Pectoralis (chest) (2 joints)
1 per side
Latissimus (back) (2 joints)
1 per side
VIDEO:
11:20
All these systems,
when do I use what?
Dependent:
On your pipeline
Result you want to achieve
The Low Level:

Animation

George Bridgman, The Human Machine 1929

The Low Level:
Animation
IK Retargetting
Look IK
Analytical IK for arms/legs
Feet on uneven surfaces
CCD IK
7 DoF for bone chains
Used for retargeting and planting
Parameterization + IK can yield
good results
IK fallback for dynamic motion
VIDEO:
12:08
Character Variation
Character Variation

System supports 3 types

of variation:
Morph-based
Attachment-based
Decal-based
Character Variation
Morph-based
Thinner/fatter body types
Change size/position of design elements
Characters divided up into ‘regions’ to
mask morph variations (vertex colors)
Attachment-based
Skin Attachments
Object Attachments
Decal-based
Small 128px dirt, scars, camouflage, etc
Layers on top of current texture
Super low resource usage
Case Study:
Variations for E3 2006
3 x 128px Camouflage Face
Textures
1 x 128px Dirt Texture
2 Head Variations
2 Body Variations
Thinner/Fatter body types, Longer/Shorter
clothing
Attachments
Skin Attachments
Hands, Heads
Object Attachments
E3 Character Variations 1/10
E3 Character Variations 2/10
E3 Character Variations 3/10
E3 Character Variations 4/10
E3 Character Variations 5/10
E3 Character Variations 6/10
E3 Character Variations 7/10
E3 Character Variations 8/10
E3 Character Variations 9/10
E3 Character Variations 10/10
Issues We Ran Into

Morph-based
Maintaining point index in LODs
(variations)
Too Much Detail? Not enough?
Designer setup time
Procedural runtime generation (examples-
based)
How much of a difference can the player
perceive ingame?
Parameterization
Non-Face attachments
Eye placement
Bone attachments
Case Study:
Character Flash
Freezing
Initial Freezing
Must be dynamic
Any character in any pose or animation
Slowly blend, do not ‘pop’ to a stop
Broken Pieces
Physicalized parts
Interactive, physicalized parts
Realistic
Actual parts
Case Study:
Character Flash
Freezing
Implementation:
Frozen Mesh
Procedural ‘Ice Shader’ (layer)
Brittle Physicalized mesh
Shattering
Ice particles emitted from joint phys proxies
Broken human bodyparts are created for most
of the bones in the character.
Pieces are spawned in the orientation of their
associated joints
Broken pieces
Left as individual physicalized entities
Pickable
VIDEO:
13:12
Baking Out
Simulations:
Isn’t That Cheating?
Baking Out
Simulations:
Isn’t That Cheating?
Same thrilling game experience
for mid-low end users (though
less interactive)
Benefits:
Musculo-skeletal systems/Organic
Deformation
Bake to joint-driven morphs
Bake to joint-driven joints
Rigid-body physics
Bake rigid body physics to keyframe data
1 joint per piece
Baked but still interactive?
Baking Out
Simulations:
Interactivity
Organics:
Pose-driven joints or morphs
Physicalization of Rigid Bodies
Use the keyframe data to generate
accurrate impulse
Physicalize the animated rigid body upon
collision
Physicalize after the animation plays
Physicalize after the joint has reached x
distance from the object center.
Physicalize when received damage
 Pre-Baked Rigid Bodies

VIDEO:
14:00
Baking Out
Simulations:
Characters at Heart
Keyframe data
Parameterization
Blend between multiple simulations
Inherent issues, but for low spec: much
better than the alternative
Compression
Can have attachments
Deformible (bending metal etc)
Can attach other characters to it
Baking Out
Simulations:
Issues We Ran Into
Draw calls
Meshes need to be merged into one
Compression
Piles of debris
Export surroundings for
external sim
Wavefront OBJ export
Terrain, objects, etc…
Can run the simulation in Max, Maya,
etc…
Better animator control for ‘Hollywood 11’
VIDEO:
14:17
Hanno Hagedorn
Senior Character Artist, Crytek

Responsibilities
Character Heads
Modeling
Texturing
Rigging
Facial Setup
Character Bodies (partially)
I am going to talk about...

Pipeline improvements
Polybump2
How we do faces, from the texture to
animation:
Asset Creation
Textures
Model
Shaders
Facial Animation System
 Crytek PolyBump2

Crunches assets with many millions of

polygons
Capable of combining multiple models, each
with many millions of polygons into one single
calculation
32/64bit 3dsmax plugin or standalone
application
Exports normal maps, displacement
maps, and ambient occlusion maps
Gives out a storage file containing 100x
the accuracy of a normal texture map
Extracting new data without recalculation
Unoccluded Area Direction

Simulates self shadowing inside the

normal map while softening hard details
Combines ACC with the normalmap
Softshadowing within the normalmap
Applies a very soft look to the skin
VIDEO:
14:40
VIDEO:
14:58
 Asset Creation

Get good texture reference

Bad texture reference
Synchronizing texture maps
Get the right software
 Get Good Texture
Reference
We all love hand painted textures
Hand painting * photorealistic / Schedule =
pain in the ass
Well lit photos are key
Unwanted lighting will plague you later on
The less lighting information will get
you closer to your desired result
 Bad texture reference

Very colored
Very blurry
Shiny nose
Lots of direct and
only few ambient
light
Very contrasted
shading
Good Texture reference

Equaly lit
Very sharp
No major highlights
Interesting details
Small
colorvariations in
the skin
Synchronizing texture
maps
Model and texture should feel like a
single unit
Pore stamps can be very handy but…
Never fit the existing texture
Create a repetitive look
Make sure your diffuse and spec
map complement your high
frequency or pore-detailed model
Base Diffuse Texture
Normalmap
Specular Map
Diffuse with ACC
lighting
Get the right Software
Z-Brush and Mudbox
Great programs for sketching out faces and
shapes
Great programs for polishing and detailing
assets
Silo is looking very promising
Combining Modeling and sculpting into one
workflow
 Shader

Rim Lighting
Sub-Surface Scattering (SSS)
Eyes
 Rim Lighting

Simulates greasy reflective Skin

Works against the classic specular look of
last-gen games
There is no such thing as ‘specular light’
in reality
Similar to Fresnel reflections
Adding ‘rim lighting’ by
Ambient color
Cubemap
Realtime automated cubemaps
No Rim Lighting
Basic Rim Lighting
Strong Rim Lighting
Sub-Surface Scattering
(SSS)
Approximates light rays penetrating the
skin and exiting another spot
Gives skin a nice reddish, fleshy look
Even simulates thin vascular tissue
SSS-map allows us to define the skins
thickness and color
No SSS
Basic SSS
Strong SSS
Eyes - Windows to the
Soul
After several tests we decided to:
Build a physically correct Eye model with
internal physiology
Works better with new shader
possibilities
Special Iris shader; glinting

3dsMax screenshot
Eyes - Windows to the
Soul
After several tests we decided to:
Build a physically correct Eye model with
internal physiology
Works better with new shader
possibilities
Special Iris shader; glinting

3dsMax screenshot
Eyes - Windows to the
Eyes - Self shadowing
Soul
After several tests we decided to:
Build a physically correct Eye model with
internal physiology
Works better with new shader
possibilities
Special Iris shader; glinting

3dsMax screenshot
Eyes - Self shadowing

Using overlaying geometry for emulating

the shaded eye
Comments Eyelids and Eyeballs together
This allows us to simulate the shadowing
of the eyelashes and surrounding bone
structure

3dsMax screenshot
Eyes - Self shadowing

Self shadowing
Using the same base as the Eyelids
saves textures and draw calls
Eyeball moves under the Overlaying
Geometry while keeping the shading

3dsMax screenshot
Eyes - Self shadowing

Self shadowing
Using the same base as the Eyelids
saves textures and draw calls
Eyeball moves under the Overlaying
Geometry while keeping the shading

3dsMax screenshot
Eyes - Self shadowing

VIDEO:
15:09
Eyes – In motion
Eyelids move along the Eyeball via
Fleshy Eyelid morphs
Automated Saccadic eye motion
Automated Eye Blinking, synced with the
Saccadic eye motion
 Eyes – In motion
Eyelids move along the Eyeball via
Fleshy Eyelid morphs
Automated Saccadic eye motion
Automated Eye Blinking, synced with the
Saccadic eye motion

VIDEO:
15:23
Facial Animation System
Facial Animation System

Muscle Morphs
Expressions
Linear
Non-linear
Phonemes
Visimes
Joystick Controls
Optical Flow
 Morphs – The Base

Muscle Morphs
89 Morphs
Each muscle a separate morph
A facial expression is sculpted by “muscle
morphs”
Asymmetrical morph setup
Natural face movement
Adds character to the facial movement
Automatic ‘fleshy eye’ movement
Eyelids move synchronized with the Eyeball
Using same topology for all faces
Easy transfer of morphs between meshes
 Expressions – The
Connection
Unlimited scalability
Core system is very simple, yet very scalable
Multiple linear and nonlinear
expressions
Can combine linear and non-linear
Setting the base for the phoneme
extraction
 Linear Expressions

Linear blending of
morphs or drivers

VIDEO:
15:34
 Non-linear expressions

Non-linear blending of morphs

Use multiple morphs to:
create a non-linear motion
keep the volume of complex or twisted shapes

VIDEO:
16:13
Expressions are drivers

System drives Morphs, Bones,

and attachments
All elements are treated equally
Can be easily combined or parameterized
 Phonemes
Phonemes
The smallest distinctive
units of speech, or the
mouth shapes to make each
sound
Strong and weak phonemes
for appropriate usage
96 phonemes, 48 strong and
48 weak
Seamless blending between
strong and weak phonemes
 Phoneme
Phonemes
The smallest distinctive
units of speech, or the
mouth shapes to make each
sound
Strong and weak phonemes
for appropriate usage
96 phonemes, 48 strong and
48 weak
Seamless blending between
strong and weak phonemes
 Visimes

Visual simplification of
phonemes
Phonemes like ‘o’, ‘u’, ‘oo’, and
‘ow’ are visually very similar
14 visimes, 7 strong, 7 weak
Massively simplifies the work
with visimes
Tweaking several phonemes at
once is handy when transferring
the setup to a different character
 Visimes

Visual simplification of
phonemes
In use by 2D animations for over
90 years
Muscle/Phoneme/Visime
Breakdown
Phoneme Extraction
Automated process
Automatically generates Phonemes from a
pure Audio file
Manual process
Breaks words into phonemes and places them
in the timeline corresponding to the audio
waveform
barely needs touching up, easy to handle,
batch-able.
VIDEO:
17:15
Joystick Controls –
Manual input device
User-friendly and intuitive
Your Grandma could animate with this!
Animator’s focus will be on animation and not
dragging sliders back and forth
Joystick Controls –
Manual input device
User-friendly and intuitive
Your Grandma could animate with this!
Animator’s focus will be on animation and not
dragging sliders back and forth
Joystick Controls –
Manual input device
Handles 4 diff expressions at once
Example, Up/Down could be the
‘jawdropper’ and ‘jaw_presser’, Left/Right
could be the ‘mouth_widener’ and
‘mouth_tightener’.
Automatically excludes opposing
morphs
No more fighting with frown/smile or
open/close
Puppeting
Real-time Mouse input capture
VIDEO:
17:28
VIDEO:
18:32
Joystick Controls –
scalability
User-definable
Joysticks plug into the cure interface, any
expressions can be mapped to them
Scalable toolset
Can be used to limit the detail an animator
sees
You can predefine A, B, and C class joystick
setups
We got inspired by Jason Osipa’s
book: ‘Stop Staring’, which
popularized joystick control setups
Optical-flow based Facial
Motion Capture
Optical-flow based Facial
Motion Capture
Getting good results with a minimum
amount of work
No expensive hardware needed
Animators here have used webcams at their
desks.
Very easy to polish
MoCap system uses the same interface the
animators do (joysticks)
Combines with the auto phoneme
extraction to polish lip-sync
Theoretical Marker Setup
Practical Marker Setup
Marker corresponding
joysticks
 Marker corresponding
joysticks

VIDEO:
19:42
The Capturing Workflow

VIDEO:
19:55
 Final Result

VIDEO:
21:11
 Final Result

Casual B-Level animation:

Used a Hand camera for Motion capturing
Used Eyeliner for the Markers
Untouched Facial Mocap Data
Automatic Phoneme Extraction
Procedural Eye blinking
Procedural Eye behavior
 Final Result

VIDEO:
21:22
Upcoming Improvements:

Giving animations to the animation

department for:
Enhancing and exagerating the acting
Polishing up Mocap Data
Cleaning up Mocap Data Keys

Improved Phoneme Blending

Improvement will come, but our mainfocus is
on finishing up the game right now ;]
Questions?
 Questions or comments?

Christopher Evans
Christopher@Crytek.de
Chris.Evans@Gmail.com

Hanno Hagedorm
Hanno@Crytek.de
Hanno.Hagedorn@Polyroyal.com

For more info,

come visit us at booth
XXXX