Notes on downloaded slides

:
The accompanying videos can be downloaded from:
www.ChrisEvans3D.com/tutorials.htm The videos were captured with various software which sometimes seriously butchers the framerate, please keep this in mind

We have added comments generally summing up what we said about each slide The audio of our session in mp3 format can be purchased from ‘GDC Radio’ (
http://store.cmpgame.com/product.php?id=1847&cat= )

Christopher Evans
Senior Technical Artist, Crytek Responsibilities:
R&D Artist Artist tools Getting characters in-game

Hanno Hagedorn
Senior Character Artist, Crytek Responsibilities
Character Heads
Modeling Texturing Rigging

Facial Setup Character Bodies (partially)

What is this stuff?
A cross section of current character-related R&D at Crytek Mostly WIP, all real-time, real assets Presenting collaborative work

Artist-driven, or real world data-driven tools

Intended Audience
Intermediate to Advanced – Requires experience and familiarity with the subject Technical Artists (TA) and artists with technical interests Not too complicated
Simple, pragmatic approaches

Session Overview
Low Level
Deformation
How you drive deformations How you deform geometry

Animation

High Level
Character Variation Case Studies Internal Tools Asset Creation Facial Editor
Setup Animation

What is a

Rejected internal concept art

What is a Character?
Person or creature?

Rejected internal concept art

What is a Character?
Vehicle or machine with moving parts?

What is a Character?
Any object or set of objects whose movement is derived from a skeleton.

What is a Character?
People and creatures Vehicles and machines Moving character attachments
Guns, backpacks, heads, hands

Dynamic, jointed physics simulations
Ropes, complex constraint setups

Pre-baked physics and broken objects
Pre-baking cinematic destruction

VIDEO: 00:04

The Low Level:

Deformatio n

George Bridgman, The Human Machine 1929

The Low Level:

Deformation
What drives the deformation?
Motion? Arbitrary inputs? Shader-based?

How do you move and deform geometry?
Is it efficient? What kind of approach?

VIDEO: 00:45

VIDEO: 00:45

The Low Level:

Deformation
What drives the deformation?
Motion? Arbitrary inputs? Shader-based? (procedural noise)

How do you move and deform geometry?
Is it efficient? What kind of approach?

Driving Deformation:

Pose Driven Systems

George Bridgman, The Human Machine 1929

Driving deformation:

Pose-Driven Systems
You can drive many things with skeletal poses:
Morph Targets Joints Maps
Outside our industry:

idea at all

Not a new

Simple 3 Vector Driver: Shape

VIDEO: 01:03

Simple Driver: Color and Shape

VIDEO: 01:09

Driving deformation:

Pose-Driven Systems
Pre-baked
Baked to keys/shapes

Runtime
In engine

Pros/Cons:

Pre-Baked Pose-Drivers
Pros:
Setup is in the asset creation app Implementations work well in most animation-based games (joints)

Cons:
Non-dynamic - Does not work with motion generated at runtime
Ragdoll physics Physically-based motion Other procedurally-generated motions

Driving deformation:

Pose-Driven Systems
Pre-baked
Baked to keys/shapes

Runtime
In engine

Driving deformation:

Runtime Pose-Drivers
Pros:
Works dynamically in any situation Only limit is the number of poses you define

Cons:
Difficult to set up Need to mirror in the asset package
Constraints Parameterization

Driving deformation:

Arbitrary-Drivers
Holy grail of driven systems General Arbitrary Drivers
General input/output system Houdini, Shadetree, etc…

The more general the better
Runtime expressions?

Not limited to deformation at all!

VIDEO: 01:16

VIDEO: 02:11

VIDEO: 02:54

VIDEO: 03:19

The Low Level:

Deformation
What drives the deformation?
Character motion? Skeletal motion? Arbitrary inputs? Shader-based? (procedural noise)

How do you move and deform geometry?
Is it efficient? Is it an Artist-centric approach?

Deformation:

Skinning

George Bridgman, The Human Machine 1929

Linear Skinning
Weight vertices in a linear fashion to n bones

Definitely nothing new First use in a game (software)
Trespasser? (1997-1998)

First use in a game (hardware)
dx8 via vert shaders dx7 rarity
Hardware ‘deceleration’

Linear Skinning,
or smooth skinning, or linear blend skinning..

Pros
Industry Standard Lots of available research

Cons
Poor volume preservation
The more a joint bends, the more it collapses

Difficult to get specific control Can be more resource intensive

Our Implementation of:

Spherical Skinning
“Hardware Skinning with Quaternions” (2004)
by Jim Hejl (EA) Game Programming Gems 4

First game to utilize it?

Pros/Cons

Spherical Skinning
Pros
Much better volume preservation Less resource usage
(Skinning position, tangent, and binormal) 72 bones per draw call
~48 (Linear w/ matrices) ~72 (Linear w/ quats)

20 instructions
~40 instructions (Linear w/ matrices) ~60 instructions (Linear w/ quats)

VIDEO: 04:11

VIDEO: 04:45

Pros/Cons

Spherical Skinning
Cons
Needs support in the asset creation package
Plugin, preview, etc…

Inherent issues with some weighting configurations
N bones per vert Non-consecutive or interleaved bones

Interleaved Bones
Example: Character Studio Biped

Pros/Cons

Spherical Skinning
Cons
Needs support in the asset creation package
Plugin, preview, etc…

Inherent issues with some weighting configurations
N bones per vert Non-consecutive or interleaved bones

The volume preservation sometimes corrects things we have come to rely on linear skinning for

Interested in New Skinning Technology?
GDC Session: Skinning with Dual Quaternions Tomorrow (Fri) 9am – 10am Room 2009, West Hall

Deformation:

Morph Targets

George Bridgman, The Human Machine 1929

Deformation:

Morphs are Your Friends
Also known as:
Blendshapes, shapes, vertex animation

Artistic approach – Artist defines the look completely
Facial animation
Sculpt muscles, phonemes, visimes

Corrective shapes
Driven by arbitrary inputs

Baked simulations
Bending metal, ocean waves, soft tissue

Deformation:

Morphs are Your Friends
Sparse morphs

Keep them light in engine
Only the verts that move Corrective shapes can be *very* light Only move the points that you have to

Sparse morphs on DX10 hardware
Sparse morphs running on dx10 hardware
Supports binding to more than 4 color images Store morphs in a texture Blend between textures

There is a demo in the DX10 SDK
They also have wrinkle maps in this example!

Driving Morphs
Pose-Driven morphs
Artist-Sculpted Pipeline
Define pose vectors Artist sculpts the pose Link sculpts to those pose vectors Parameterize/blend between poses

Simulation-Sculpted
Artist creates deformation Links deformation to some arbitrary parameter in the engine

Driving Morphs
Our asset creation tool
Allows the artist to pose the rigged character
We decided to use pre-defined pose vectors

Generates a mesh in that pose for him to sculpt Extracts the artists sculpt with the skin deformation culled Created a shape for that pose

VIDEO: 05:26

VIDEO: 06:43

Our In Engine Setup
Apply the shape locally before the skinning Assign the shape to the correct bone and pose vector This is the simplest scenario
Single pose vector

VIDEO: 06:49

This is a simple test, there are many more uses!
Corrective morph shapes

SAY GOODBYE TO SKINNING ARTIFACTS!
Complex deformation
You don’t have to sculpt these targets by hand! The sky is the limit

VIDEO: 09:14

VIDEO: 10:35

Deformation:

Helper Joints

George Bridgman, The Human Machine 1929

Helper Joints
Jason Parks: Advanced Deformations on Runtime Characters – GDC 05/06
http://www.jason-parks.com/HelperJoints/ http://www.jason-parks.com/MuscleSystems/

Great but less artist-driven
Universal Needs a lot of technical support
Non-intuitive

Examples

Same driving issues as with morphs
Pre-baked vs. Runtime

VIDEO: 10:46

Example: Shark

Example: Nanosuit

Example: Nanosuit

Example: Nanosuit

Helper Joints (12):
Deltoids (shoulders) (8 joints)
4 on each shoulder

Pectoralis (chest) (2 joints)
1 per side

Latissimus (back) (2 joints)
1 per side

VIDEO: 11:20

All these systems, when do I use what?
Dependent:
On your pipeline Result you want to achieve

The Low Level:

Animation

George Bridgman, The Human Machine 1929

The Low Level: Animation
IK Retargetting
Look IK
Analytical IK for arms/legs

Feet on uneven surfaces CCD IK
7 DoF for bone chains Used for retargeting and planting

Parameterization + IK can yield good results
IK fallback for dynamic motion

VIDEO: 12:08

Character Variation

Character Variation
System supports 3 types of variation:
Morph-based Attachment-based Decal-based

Character Variation
Morph-based
Thinner/fatter body types Change size/position of design elements Characters divided up into ‘regions’ to mask morph variations (vertex colors)

Attachment-based
Skin Attachments Object Attachments

Decal-based
Small 128px dirt, scars, camouflage, etc Layers on top of current texture
Super low resource usage

Case Study:

Variations for E3 2006
3 x 128px Camouflage Face Textures 1 x 128px Dirt Texture 2 Head Variations 2 Body Variations
Thinner/Fatter body types, Longer/Shorter clothing

Attachments
Skin Attachments
Hands, Heads

Object Attachments

E3 Character Variations 1/10

E3 Character Variations 2/10

E3 Character Variations 3/10

E3 Character Variations 4/10

E3 Character Variations 5/10

E3 Character Variations 6/10

E3 Character Variations 7/10

E3 Character Variations 8/10

E3 Character Variations 9/10

E3 Character Variations 10/10

Issues We Ran Into
Morph-based
Maintaining point index in LODs (variations)

Too Much Detail? Not enough?
Designer setup time
Procedural runtime generation (examplesbased)

How much of a difference can the player perceive ingame?

Parameterization
Non-Face attachments
Eye placement Bone attachments

Case Study:

Character Flash Freezing
Initial Freezing
Must be dynamic
Any character in any pose or animation Slowly blend, do not ‘pop’ to a stop

Broken Pieces
Physicalized parts
Interactive, physicalized parts

Realistic
Actual parts

Case Study:

Character Flash Freezing
Implementation:
Frozen Mesh
Procedural ‘Ice Shader’ (layer) Brittle Physicalized mesh

Shattering
Ice particles emitted from joint phys proxies Broken human bodyparts are created for most of the bones in the character. Pieces are spawned in the orientation of their associated joints

Broken pieces
Left as individual physicalized entities Pickable

VIDEO: 13:12

Baking Out Simulations:
Isn’t That Cheating?

Baking Out Simulations:
Isn’t That Cheating?
Same thrilling game experience for mid-low end users (though less interactive) Benefits:
Musculo-skeletal systems/Organic Deformation
Bake to joint-driven morphs Bake to joint-driven joints

Rigid-body physics
Bake rigid body physics to keyframe data
1 joint per piece

Baked but still interactive?

Baking Out Simulations:
Interactivity
Organics:
Pose-driven joints or morphs

Physicalization of Rigid Bodies
Use the keyframe data to generate accurrate impulse Physicalize the animated rigid body upon collision Physicalize after the animation plays Physicalize after the joint has reached x distance from the object center. Physicalize when received damage

Pre-Baked Rigid Bodies

VIDEO: 14:00

Baking Out Simulations:
Characters at Heart
Keyframe data
Parameterization
Blend between multiple simulations Inherent issues, but for low spec: much better than the alternative

Compression Can have attachments
Deformible (bending metal etc) Can attach other characters to it

Baking Out Simulations:
Issues We Ran Into

Draw calls
Meshes need to be merged into one

Compression Piles of debris Export surroundings for external sim
Wavefront OBJ export
Terrain, objects, etc…

Can run the simulation in Max, Maya, etc…
Better animator control for ‘Hollywood 11’

VIDEO: 14:17

Hanno Hagedorn
Senior Character Artist, Crytek Responsibilities
Character Heads
Modeling Texturing Rigging

Facial Setup Character Bodies (partially)

I am going to talk about...
Pipeline improvements
Polybump2

How we do faces, from the texture to animation:
Asset Creation
Textures Model

Shaders Facial Animation System

Crytek PolyBump2
Crunches assets with many millions of polygons
Capable of combining multiple models, each with many millions of polygons into one single calculation

32/64bit 3dsmax plugin or standalone application Exports normal maps, displacement maps, and ambient occlusion maps Gives out a storage file containing 100x the accuracy of a normal texture map
Extracting new data without recalculation

Unoccluded Area Direction
Simulates self shadowing inside the normal map while softening hard details Combines ACC with the normalmap
Softshadowing within the normalmap Applies a very soft look to the skin

VIDEO: 14:40

VIDEO: 14:58

Asset Creation
Get good texture reference Bad texture reference Synchronizing texture maps Get the right software

Get Good Texture Reference
We all love hand painted textures
Hand painting * photorealistic / Schedule = pain in the ass

Well lit photos are key
Unwanted lighting will plague you later on

The less lighting information will get you closer to your desired result

Bad texture reference
Very colored Very blurry Shiny nose Lots of direct and only few ambient light Very contrasted shading

Good Texture reference
Equaly lit Very sharp No major highlights Interesting details Small colorvariations in the skin

Synchronizing texture maps
Model and texture should feel like a single unit Pore stamps can be very handy but…
Never fit the existing texture Create a repetitive look

Make sure your diffuse and spec map complement your high frequency or pore-detailed model

Base Diffuse Texture

Normalmap

Specular Map

Diffuse with ACC lighting

Get the right Software
Z-Brush and Mudbox
Great programs for sketching out faces and shapes Great programs for polishing and detailing assets

Silo is looking very promising
Combining Modeling and sculpting into one workflow

Shader
Rim Lighting Sub-Surface Scattering (SSS) Eyes

Rim Lighting
Simulates greasy reflective Skin Works against the classic specular look of last-gen games There is no such thing as ‘specular light’ in reality Similar to Fresnel reflections Adding ‘rim lighting’ by
Ambient color Cubemap Realtime automated cubemaps

No Rim Lighting

Basic Rim Lighting

Strong Rim Lighting

Sub-Surface Scattering (SSS)
Approximates light rays penetrating the skin and exiting another spot Gives skin a nice reddish, fleshy look
Even simulates thin vascular tissue

SSS-map allows us to define the skins thickness and color

No SSS

Basic SSS

Strong SSS

Eyes - Windows to the Soul
After several tests we decided to:
Build a physically correct Eye model with internal physiology Works better with new shader possibilities Special Iris shader; glinting

3dsMax screenshot

Eyes - Windows to the Soul
After several tests we decided to:
Build a physically correct Eye model with internal physiology Works better with new shader possibilities Special Iris shader; glinting

3dsMax screenshot

Eyes - Windows to the Eyes - Self shadowing Soul
After several tests we decided to:
Build a physically correct Eye model with internal physiology Works better with new shader possibilities Special Iris shader; glinting

3dsMax screenshot

Eyes - Self shadowing
Using overlaying geometry for emulating the shaded eye Comments Eyelids and Eyeballs together This allows us to simulate the shadowing of the eyelashes and surrounding bone structure

3dsMax screenshot

Eyes - Self shadowing
Self shadowing
Using the same base as the Eyelids saves textures and draw calls Eyeball moves under the Overlaying Geometry while keeping the shading

3dsMax screenshot

Eyes - Self shadowing
Self shadowing
Using the same base as the Eyelids saves textures and draw calls Eyeball moves under the Overlaying Geometry while keeping the shading

3dsMax screenshot

Eyes - Self shadowing

VIDEO: 15:09

Eyes – In motion
Eyelids move along the Eyeball via Fleshy Eyelid morphs Automated Saccadic eye motion Automated Eye Blinking, synced with the Saccadic eye motion

Eyes – In motion
Eyelids move along the Eyeball via Fleshy Eyelid morphs Automated Saccadic eye motion Automated Eye Blinking, synced with the Saccadic eye motion

VIDEO: 15:23

Facial Animation System

Facial Animation System
Muscle Morphs Expressions
Linear Non-linear Phonemes Visimes

Joystick Controls Optical Flow

Morphs – The Base
Muscle Morphs
89 Morphs
Each muscle a separate morph A facial expression is sculpted by “muscle morphs”

Asymmetrical morph setup
Natural face movement Adds character to the facial movement

Automatic ‘fleshy eye’ movement
Eyelids move synchronized with the Eyeball

Using same topology for all faces
Easy transfer of morphs between meshes

Expressions – The Connection
Unlimited scalability
Core system is very simple, yet very scalable

Multiple linear and nonlinear expressions
Can combine linear and non-linear

Setting the base for the phoneme extraction

Linear Expressions
Linear blending of morphs or drivers

VIDEO: 15:34

Non-linear expressions
Non-linear blending of morphs Use multiple morphs to:
create a non-linear motion keep the volume of complex or twisted shapes

VIDEO: 16:13

Expressions are drivers
System drives Morphs, Bones, and attachments
All elements are treated equally Can be easily combined or parameterized

Phonemes
Phonemes
The smallest distinctive units of speech, or the mouth shapes to make each sound
Strong and weak phonemes for appropriate usage
96 phonemes, 48 strong and 48 weak

Seamless blending between strong and weak phonemes

Phoneme
Phonemes
The smallest distinctive units of speech, or the mouth shapes to make each sound
Strong and weak phonemes for appropriate usage
96 phonemes, 48 strong and 48 weak

Seamless blending between strong and weak phonemes

Visimes
Visual simplification of phonemes
Phonemes like ‘o’, ‘u’, ‘oo’, and ‘ow’ are visually very similar 14 visimes, 7 strong, 7 weak Massively simplifies the work with visimes Tweaking several phonemes at once is handy when transferring the setup to a different character

Visimes
Visual simplification of phonemes
In use by 2D animations for over 90 years

Muscle/Phoneme/Visime Breakdown

Phoneme Extraction
Automated process
Automatically generates Phonemes from a pure Audio file

Manual process
Breaks words into phonemes and places them in the timeline corresponding to the audio waveform barely needs touching up, easy to handle, batch-able.

VIDEO: 17:15

Joystick Controls – Manual input device
User-friendly and intuitive
Your Grandma could animate with this! Animator’s focus will be on animation and not dragging sliders back and forth

Joystick Controls – Manual input device
User-friendly and intuitive
Your Grandma could animate with this! Animator’s focus will be on animation and not dragging sliders back and forth

Joystick Controls – Manual input device
Handles 4 diff expressions at once
Example, Up/Down could be the ‘jawdropper’ and ‘jaw_presser’, Left/Right could be the ‘mouth_widener’ and ‘mouth_tightener’.

Automatically excludes opposing morphs
No more fighting with frown/smile or open/close

Puppeting
Real-time Mouse input capture

VIDEO: 17:28

VIDEO: 18:32

Joystick Controls – scalability
User-definable
Joysticks plug into the cure interface, any expressions can be mapped to them

Scalable toolset
Can be used to limit the detail an animator sees You can predefine A, B, and C class joystick setups

We got inspired by Jason Osipa’s book: ‘Stop Staring’, which popularized joystick control setups

Optical-flow based Facial Motion Capture

Optical-flow based Facial Motion Capture
Getting good results with a minimum amount of work No expensive hardware needed
Animators here have used webcams at their desks.

Very easy to polish
MoCap system uses the same interface the animators do (joysticks)

Combines with the auto phoneme extraction to polish lip-sync

Theoretical Marker Setup

Practical Marker Setup

Marker corresponding joysticks

Marker corresponding joysticks

VIDEO: 19:42

The Capturing Workflow

VIDEO: 19:55

Final Result

VIDEO: 21:11

Final Result
Casual B-Level animation:
Used a Hand camera for Motion capturing Used Eyeliner for the Markers Untouched Facial Mocap Data Automatic Phoneme Extraction Procedural Eye blinking Procedural Eye behavior

Final Result

VIDEO: 21:22

Upcoming Improvements:
Giving animations to the animation department for:
Enhancing and exagerating the acting Polishing up Mocap Data Cleaning up Mocap Data Keys

Improved Phoneme Blending
Improvement will come, but our mainfocus is on finishing up the game right now ;]

Questions?

Questions or comments? Christopher Evans Christopher@Crytek.de Chris.Evans@Gmail.com Hanno Hagedorm Hanno@Crytek.de Hanno.Hagedorn@Polyroyal.com For more info, come visit us at booth XXXX