Emotion Detection With Graphic Visualization

Katie McNabb Final Project CS585

The Goal:
"Empathetic Painting: Interactive Stylization Through Observed Emotional State" Shugrina, Betke, Collomosse. Two Priorities
Capture emotions on a continuum Integrate emotion detection with interactive visualization

The paper's approach

8 Action Units based on Ekman's AUs

The paper's approach cont.

Using normalization of these 8 action units we get an "emotion" value that lands on this matrix:

Papers approach: Graphic Visualization

Paint based on emotion values:

My approach: facial action units

4 facial action units:
Action Units Eyebrow raised Eyebrow furrowed Smile Frown Pleasure .5 -.5 1 -1 Arousal .7 -.7 0 0

My approach: emotion matrix

A modified emotion matrix:

My approach: image filter

Smile correlates to image brightness
Smile - high brightness Frown - low brightness

Eyebrow correlates to contrast

Eyebrow raise - high contrast Eyebrow furrow - low contrast

My approach to detecting facial expression

Track facial features with template matching OpenCV function - cvMatchTemplate Track change in distance from nose

My facial feature detection

Four distances from the nose were used:

left mouth, right mouth, left eye, and right eye

Results
Frown:

Results
Frown and eye raised:

Graphic Visualization
Original Image

Graphic Visualization
Smile Frown and Eye Raise

Results
22 experiments: Correct results: 16 Incorrect results: 6

Takeaways:
Not all facial expressions are created equal Eye raises much better than mouth movement

Future work
Improve robustness of feature tracking Find other metrics for detecting facial action Image filtering techniques

Sources
Shugrina, Betke, & Collomosse. (2006). Empathetic Painting: Interactive Stylization through observed emotional state. Proceedings of the 4th international symposium on Non-photorealistic animation and rendering, 8796. OpenCV Eye Tracking http://nashruddin.com/eyetracking-track-user-eye. html.