Storytelling and Understanding AI: Toward A Critical Technical Practice

Storytelling and Understanding AI
Toward A Critical Technical Practice

Invited Colloquium, Genesis GroupMIT
rogelio@cs.utah.edu
Rogelio E. Cardona-Rivera, Ph.D. http://rogel.io
Assistant Professor @recardona

School of Computing, Entertainment Arts and Engineering Program
University of Utah
04 09 2021
qed.cs.utah.edu LABORATORY FOR QUANTITATIVE EXPERIENCE DESIGN

We have no clue
We have no clue why stories are so
effective,
effective, how to intentionally
structure them,
structure them,
nor how they impact us
structure them,
nor how they impact us
Idea: Computational Narrative Models can help

What is a Computational Model?

A Function.
The Introspection Age
The Rational Reconstruction Age <latexit sha1_base64="N/gMg3lJTkBhnJfOXczuRg6YvPE=">AAACAnicbVDLSgNBEJyNrxhfUY9eBoMQL2FXRD0GvHiMYh6QLGF2MpsMmZ1ZZnrFsOTm2at+gzfx6o/4Cf6Fs8keNLGgoajqprsriAU34LpfTmFldW19o7hZ2tre2d0r7x+0jEo0ZU2qhNKdgBgmuGRN4CBYJ9aMRIFg7WB8nfntB6YNV/IeJjHzIzKUPOSUQCaF1cfTfrni1twZ8DLxclJBORr98ndvoGgSMQlUEGO6nhuDnxINnAo2LfUSw2JCx2TIupZKEjHjp7Nbp/jEKgMcKm1LAp6pvydSEhkziQLbGREYmUUvE//1MgWUEmbhAAiv/JTLOAEm6Xx/mAgMCmd54AHXjIKYWEKo5vYFTEdEEwo2tZLNxltMYpm0zmreRc27Pa/U7/KUiugIHaMq8tAlqqMb1EBNRNEIPaMX9Oo8OW/Ou/Mxby04+cwh+gPn8wcD+5fD</latexit>
The Embodied and Cognitive Age f (x)

A Function.

Inpu
t
A Function.

Inpu
t
Outp
ut
A Function.

Beh Inpu
avio t
Outp r
ut
A Function.
Inpu
t
<latexit sha1_base64="N/gMg3lJTkBhnJfOXczuRg6YvPE=">AAACAnicbVDLSgNBEJyNrxhfUY9eBoMQL2FXRD0GvHiMYh6QLGF2MpsMmZ1ZZnrFsOTm2at+gzfx6o/4Cf6Fs8keNLGgoajqprsriAU34LpfTmFldW19o7hZ2tre2d0r7x+0jEo0ZU2qhNKdgBgmuGRN4CBYJ9aMRIFg7WB8nfntB6YNV/IeJjHzIzKUPOSUQCaF1cfTfrni1twZ8DLxclJBORr98ndvoGgSMQlUEGO6nhuDnxINnAo2LfUSw2JCx2TIupZKEjHjp7Nbp/jEKgMcKm1LAp6pvydSEhkziQLbGREYmUUvE//1MgWUEmbhAAiv/JTLOAEm6Xx/mAgMCmd54AHXjIKYWEKo5vYFTEdEEwo2tZLNxltMYpm0zmreRc27Pa/U7/KUiugIHaMq8tAlqqMb1EBNRNEIPaMX9Oo8OW/Ou/Mxby04+cwh+gPn8wcD+5fD</latexit>
f (x) Output
The Rational Reconstruction Age
Behavio
The Embodied and Cognitive Age
r
A Function.
Inpu
The Introspection Age ? t
Output
f (x)
Behavio
r
A Function.
Inpu
The Introspection Age Knowledge t
Output
f (x) Representation
Behavio
r
A Function.
Inpu
Output
Behavio
r
What is a Computational Model of Narrative?
A Function.
Inpu
Output
Behavio
r
What is a Computational Model of Narrative?
How do we build them?
The Storytelling AI Frontier Needs People
A Historical, Biased Perspective on Building Narrative Models

“May I say that I have not thoroughly enjoyed

serving with humans? I find their illogic and foolish
emotions a constant irritant.”
“What seems necessary for
storytelling and
understanding?”
storytelling and
understanding?”
The Need for Knowledge
Lebowitz, Michael. "Story-telling as planning and learning." Poetics 14.6 (1985): 483-502.
storytelling and
understanding?”
The Centrality of Planning
Meehan, James R. "TALE-SPIN, An Interactive Program that Writes Stories." IJCAI. 1977.
Storytelling AI
storytelling and
understanding?”
The Centrality of Planning
Story Understanding AI
By the 1990s, we knew…
Wilensky et al.—
people predict which goals and subsequent plans explain observed
actions by characters
Norvig—
scripts are important for generating knowledge-based inferences
Mueller—
spatiotemporal reasoning constrains story inferences
Lehnert—
means-ends (causal) and hierarchical (purposive) reasoning contribute
Black and Bower— the most to a person’s memory of a story
Winston—
hierarchical problem solving is key for inferencing and understanding
it is possible to combine these in a principled manner
The Rational
Reconstruction Age
“You've shown none of the concern that Captain

Picard would for the safety of his ship, the welfare
of his crew.”
“What do others tell us
is important?”
The Turn to:
Narratology
Cognitive Science
Design
Narratively
IntelligentGame
AI
Type a quote
here.
Psyc
holo
gy
AI+HCI
Narratology
Design
An example agenda in the

Johnny Appleseed
Science of Narrative
What is Narrative Intelligence?
Unique human capacity to

understand our environment in
terms of stories
(Heider and Simmel, 1944)

Why Narrative Intelligence matters
Narrative framing makes interaction more compelling


Entertainment video games

Entertainment training simulations
Education

Entertainment gamification
Education
Engagement

Entertainment
Education
Engagement
Difficult to engineer
AI may help ameliorate authorial burden
Interactive Narrative (IN)
AI+
HCI
Mediates actions
through a narrative
framing
Entertainment
Education
Engagement
Difficult to engineer
AI may help ameliorate authorial burden
AI+
HCI
Mediates actions
through a narrative
framing
Abstraction of
story as
trajectory of
Story Director
world states
Authorswith Narratives as
plans
Designer
Narratives as Plans
AI+
HCI
Story generation as a classical planning problem P = hsi , g, Di

si: initial state
g : goal conditions
: set of (domain) actions, predicates, and objects
D
Search for sequence to transform →
si g
Plans and planning in narrative generation: a review of plan-based approaches to the generation of story, discourse and
interactivity in narratives (Young et al., 2013)
Narratives as Plans
AI+
HCI
Actions encoded as template operators
Planning Domain Definition Language
(:action pick-up :parameters (?agent ?item ?location)

:precondition (and (at ?item ?location) (at ?agent ?location))
:effect (and (not (at ?item ?location)) (has ?agent ?item)))
Narratives as Plans
AI+
HCI

PDDL expanded with consenting agents

:effect (and (not (at ?item ?location)) (has ?agent ?item))
:agents (?agent))
What is the
Narratives as Plans problem here?

PDDL expanded with consenting agents

:effect (and (not (at ?item ?location)) (has ?agent ?item))
:agents (?agent))
` AI+
HCI
Solution to a planning problem is a

plan
P = hs , g, Di ⇡ = hS, B, Li
i
Automated Planning AI+
HCI

plan
P = hs , g, Di ⇡ = hS, B, Li
i
: steps
Pick-up Pick-up
S si g
Disenchant
HCI

plan
P = hs , g, Di ⇡ = hS, B, Li
i
: steps
Pick-up Pick-up
S: bindings si g
B Disenchant
HCI

plan
P = hs , g, Di ⇡ = hS, B, Li
i
: steps
Pick-up Pick-up
S: bindings si g
B: causal links(e.g. Disenchant )
L
hs1 , , s2 i
(has ARTHUR SPELLBOOK)
Example: A Knight’s Tale AI+
NarratologyHCI
NarratologyHCI
(define (domain KNIGHT)

(:requirements :strips)
(:predicates
(at ?x ?y) (has ?x ?y) (path ?x ?y) (asleep ?x)
(enchanted ?x))
(:action pick-up :parameters (?agent ?item ?
location) …)
(:action move :parameters (?agent ?from ?to) …)
(:action disenchant :parameters (?agent ?obj ?
location ?book) …)
(:action wake-up :parameters (?agent ?sleeper ?
location) …) )
NarratologyHCI
(define (domain KNIGHT) (define (problem STORY) (:domain

(:requirements :strips) KNIGHT)
(:predicates (:objects
(at ?x ?y) (has ?x ?y) (path ?x ?y) (asleep ?x) ARTHUR MERLIN SPELLBOOK
(enchanted ?x)) MERLINBOOK EXCALIBUR
(:action pick-up :parameters (?agent ?item ? FOREST HOME)
location) …) (:init
(:action move :parameters (?agent ?from ?to) …) (at ARTHUR FOREST)
(:action disenchant :parameters (?agent ?obj ? (at MERLIN FOREST)
location ?book) …) (has MERLIN MERLINBOOK)
(:action wake-up :parameters (?agent ?sleeper ? (asleep MERLIN) (at SPELLBOOK
location) …) ) FOREST)
(at EXCALIBUR FOREST)
(enchanted EXCALIBUR) (path
FOREST HOME))
(:goal
(has ARTHUR EXCALIBUR))
NarratologyHCI
(define (problem STORY) (:domain

KNIGHT)
(:objects
ARTHUR MERLIN SPELLBOOK
MERLINBOOK EXCALIBUR
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
(has MERLIN MERLINBOOK)
(asleep MERLIN) (at SPELLBOOK
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
FOREST HOME))
(:goal
NarratologyHCI

KNIGHT)
(:objects
FOREST HOME)
(:init
(at ARTHUR FOREST)
(at MERLIN FOREST)
FOREST)
Pick-up Pick-up
si g FOREST HOME))
(:goal
Disenchant
AI+
NarratologyHCI
Story Director
Authorswith
Designer
AI+
NarratologyHCI
si g
Story Director
Authorswith
Designer
AI+
NarratologyHCI
si g Interactswith
Play
Story Director
Authorswith
Designer
AI+
NarratologyHCI
si g Interactswith
Play
Story Director Player can act as afforded by

the logical world state
Authorswith
Designer
IN Play as Game Tree Search AI+
NarratologyHCI
Pick-up Pick-up
si g
Disenchant
Intended Narrative Plan

NarratologyHCI
Pick-up Disenchant Pick-up

si g
Chronology — Player & System take turns
On System Turn: Advance Narrative Agenda
On Player Turn: ???

NarratologyHCI

si g
On Player Turn: ???

NarratologyHCI

si g
Move
On Player Turn: ???

NarratologyHCI

si g
Move
Wake-up
NarratologyHCI

si p g
Move
q
Wake-up Many trajectories
r
NarratologyHCI

si p g
Move
…
q …
…
…
Wake-up Many many

r trajectories
NarratologyHCI

si p g
Move
…
q …
…
…
Wake-up Many many

r trajectories
Not all are good
NarratologyHCI

si p g
Move
…
q g
…
…
is unreachable
Wake-up Many many

r trajectories
Not all are good
NarratologyHCI

si p g
Move
…
q g
…
…
is unreachable
Wake-up Many many

r trajectories
Not all are good
Mediator is needed
AI+
NarratologyHCI
si g Interactswith
Play
Story Director
Authorswith
Designer
AI+
NarratologyHCI
si g Interactswith
Mediator Play
Story Director
Authorswith
Designer
AI+
NarratologyHCI
si g Interactswith
Mediator Play
Story Director
Director & Mediator
collaborate
Accept
Authorswith
Re-plan around
Designer Fail user actions
AI+
NarratologyHCI

si p g
Move
…
q …
…
…
Wake-up
r
Why would players pick these? AI+
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
si g
… …
… …
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
si g
… …
… …
Valued as
completions
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
si g
… …
… …
Valued as
completions
If often, represents:
More work for system
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
si g
… …
… …
Valued as completions
If often, represents:
More work for system
Failure of design
Automated Design Problem AI+
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
…
…
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
Managing the player’s

intent, which fluctuates
due to narrative
intelligence
…
…
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design

due to narrative
intelligence
Comprehension …
…
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design

due to narrative
intelligence
Comprehension …
…
Role-play
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design

due to narrative
intelligence
Comprehension …
…
Role-play
Desire for Agency

NarratologyHCI
Ps
yc
h
ol
o
g
y
Design

due to narrative
intelligence
Comprehension …
…
Role-play
Desire for Agency
…
Example Science of Game Design

due to narrative
intelligence
Comprehension …
…
Role-play
Desire for Agency

Example Science of Storygame Design

due to narrative
intelligence
Comprehension …
…
Role-play
Desire for Agency

Managing the player’s intent, which fluctuates due to

narrative intelligence
Comprehension
Role-play …
…
Desire for Agency
In the context of the Automated Design Problem


Comprehension
Role-play …
…
Desire for Agency

Modeling Story Understanding
Ps
yc
h
ol
o
g
y
Narratology
Readers as problem
solvers (Gerrig and Bernardo, 1994)
Question Answering in the Context of Stories Generated by Computers(Cardona-Rivera, Price, Winer, and Young,
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Readers as problem
Planning is a model of
problem solving (Tate, 2001)
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Readers as problem
Planning is a model of
problem solving (Tate, 2001)
Idea: narrative plan as a
proxy for mental state
2016)
Understanding as Planning
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
The QUEST Model of Comprehension

Comprehension as Q&A (Graesser and Franklin, 1990)
Predicts normative answers to questions

Why? How? When? What enabled?
What was the consequence?
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle
The QUESTText Graph
Arthur Arthur
wants wants
disenchante Reason
Excalibur
d Goal

Arthur
Comprehension as Q&A
Outcome
disenchants
Excalibur
Event
Why? How? When?
ExcaliburWhat enabled?
disenchante
Consequence
d
State
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Example QUEST “Why?” Search
Arthur Arthur
wants wants
disenchante Reason
Excalibur
d Goal

Arthur
Why did Arthur
Comprehension
Outcome
disenchants as Q&A disenchant
Excalibur
Excalibur?
Event
Why? How? When?
disenchante
Consequence
d
State
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Arthur Arthur
wants wants
disenchante Reason
Excalibur
d Goal

Arthur
Why did Arthur
Comprehension
Outcome
Excalibur
Excalibur?
Event
Why? How? When?
disenchante
Consequence
d
State
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Arthur Arthur
wants wants
disenchante Reason
Excalibur
d Goal

Arthur
Why did Arthur
Comprehension
Outcome
Excalibur
Excalibur?
Event
Why? How? When?
disenchante
Consequence
d
State
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Arthur Arthur
wants wants
disenchante Reason
Excalibur
d Goal

Arthur
Why did Arthur
Comprehension
Outcome
Excalibur
Excalibur?
Event
Why? How? When?
disenchante
Consequence
d
State
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Arthur Arthur
wants wants
disenchante Reason
Excalibur
d Goal

Candidate Answers Why did Arthur
Comprehension as Q&A disenchant
Excalibur?
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle
Plan to QUEST
Text Graph Mapping Algorithm
Body Level One
⇡
Given a plan = hS, B, Li :
1.
Body Level Two B
8s 2 S , generate event node ei with
a.
8effects e 2 S , generate
Body Level Three state node ti with
B
Connect Consequence Arcs for all ti → ei , ei →ti+1 in
2.
L
Body
For all Level
literalsFour
in
L, generate goal node Bli with
3.
Connect Reason Arcs for all goal nodes, by
Body Level Five
4.
Why?
ancestry How? When? What enabled?
5.
What was the consequence? L
Connect Outcome Arcs for all li →ei in
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle
Plan to QUEST
Text Graph Mapping Algorithm
Body Level One
Mapping
Given data
⇡
a plan structure
= hS, B, Li semantics
: to
Body
cognitive Level Two
semantics
8s 2 S , generate
1. B
event node ei with
8effects
a.
e 2 S , generate
Body Level Three state node ti with
B
Connect Consequence Arcs for all ti → ei , ei →ti+1 in
2.
L
Body
For all Level
literalsFour
in
L, generate goal node Bli with
3.
Connect Reason Arcs for all goal nodes, by
Body Level Five
4.
Why?
ancestry How? When? What enabled?
5.
What was the consequence? L
Connect Outcome Arcs for all li →ei in
2016)
Understanding as Planning AI+
NarratologyHCI
Ps
yc
h
ol
o
g
y
Design
Subtitle Text
Evaluating the Mapping
Replicated QUEST
Validation experiment
(Graesser, Lang, and Roberts, 1991)
Original: manual graph

Ours: generated graph
Participants gave goodness-

of-answer Likert data for
Q&A pairs
Predicted their answers
Strong support for model
(N=695)
2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Takeaway
Pick-up Pick-up
si g
Disenchant

2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Takeaway
Pick-up Pick-up
si g
Disenchant
ComprehensionGeneration
as Q&A

2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Design
Subtitle Text
Takeaway
Pick-up Pick-up
si g
Disenchant
ComprehensionGeneration
as Q&A Comprehension

2016)

Comprehension
Role-play …
…
Desire for Agency


Comprehension
Role-play
Role-play …
…
Desire for Agency

Determinants of Player Choice AI+
NarratologyHCI
Ps
yc
h
ol
o
g
y
Tripartite Model of
Player Behavior
(Waskul and Lusk, 2004)
Person
Player
Persona (Roles)
The Mimesis Effect(Domínguez, Cardona-Rivera, Vance and Roberts, 2016)

Role-play as Preferred Actions
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Roles
Fighter
Wizard
Rogue
Participants (n=210)
played 1-of-3 games
Assigned Role (78)
Chosen Role (91)
No Explicit Role
(41) The Mimesis Effect(Domínguez, Cardona-Rivera, Vance and Roberts, 2016)
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Mimesis Effect
Players prefer to act as
expected from
assigned/chosen role
Players with no
explicit role self-select
and remain consistent

Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
expected from
assigned/chosen role.
Players with no
and remain consistent Chronology

Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
expected from
Players with no

Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
expected from
Players with no

Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
expected from
Players with no
Inferences

Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
expected from
Players with no
Inferences

Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
expected from
Preferred!
Players with no


Comprehension
Role-play
Role-play …
…
Desire for Agency


Comprehension
Desire for Agency
Role-play …
…
Desire for Agency

Pursuing Greater Agency Narratology
Ps
yc
h
ol
o
g
y
Satisfying power to take

meaningful action and
see the results of our
decisions & choices
(Murray 1997)
What is meaningful?
The effect of
feedback
or
Some choices
were “greater v.
agency” ones or
Achieving the Illusion of Agency(Fendt, Harrison, Ware, Cardona-Rivera and Roberts, 2012)
🥇BEST PAPER, ICIDS2012

Foreseeing Meaningful Choices AI+
NarratologyHCI
Ps
yc
h
ol
o
g
y
Idea: Greater agency — greater difference

(greater meaning)
Method: Measure choice story outcomes
Formalize story content
Define story content difference
Compare choices through story content
difference
Foreseeing Meaningful Choices(Cardona-Rivera, Robertson, Ware, Harrison, Roberts, and Young, 2014)
A Formalism of Story Content Narratology
Ps
yc
h
ol
o
g
y
The Event-Indexing Model (Zwaan, Langston, Graesser 1995)
picks up disenchants picks up
Consumers “chunk” story information into

events
space space space
time time time
causal causal causal
goals goals goals
entities entities entities
Story Content Difference AI+
NarratologyHCI
Ps
yc
h
ol
o
g
y
picks up
Situation Vector
space
time
causal
goals
entities
NarratologyHCI
Ps
yc
h
ol
o
g
y
picks up
Situation Vector
forest
time point 3
primary
wants excalibur
arthur, excalibur
NarratologyHCI
Ps
yc
h
ol
o
g
y
picks up
Situation Vector
Change Function
: SV ! [0, 5]
forest
time point 3
primary
wants excalibur
arthur, excalibur
NarratologyHCI
Ps
yc
h
ol
o
g
y
picks up picks up
Situation Vector
Change Function
: SV ! [0, 5]
forest forest
time point 1 time point 3
primary primary
wants excalibur wants excalibur
arthur, spellbook arthur, excalibur
NarratologyHCI
Ps
yc
h
ol
o
g
y
picks up picks up
Situation Vector
Change Function
: SV ! [0, 5]
forest forest
time point 1 time point 3
primary primary
wants excalibur wants excalibur
arthur, spellbook =2 arthur, excalibur
Agency as Function of Outcomes
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Participants (N=88) played custom CYOA

6 binary choices
Answered 5-point Likert prompts for agency

Page Trend Test supports our theory
=0 6
=
(Vermeulen et al. 2010)
0
H0 : M d C 0 = M d C5 = M d C3 = M d C1 = M d C2 = M d C1
HA : M d C 0 < M d C 5 < M d C3 < M d C1 < M d C2 < M d C1
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
Participants (N=88)
played custom CYOA
6 binary choices
Answered 5-point Likert

prompts for agency Chronology
Page Trend Test
supports our theory
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
Participants (N=88)
played custom CYOA
6 binary choices

Page Trend Test Inferences
supports our theory
Ps
yc
h
ol
o
AI+ g
NarratologyHCI
y
Subtitle Text
Takeaway
Participants (N=88)
played custom CYOA
6 binary choices
⇢ ,agency
Page Trend Test
supports our theory

Comprehension
Desire for Agency
Role-play …
…
Desire for Agency


Comprehension
Role-play …
…
Desire for Agency

The Embodied and
Cognitive Age
A Function.

Beh Inpu
avio t
Outp r
ut


Beh
avio
r
How?
Why?
For Whom?
f (x)
Beh
avio
r
The Storytelling AI Frontier Needs People
A Historical, Biased Perspective on Building Narrative Models
The Introspection Age Recently awarded a CAREER!

rogelio@cs.utah.edu

University of Utah
End
A Science of Game Design
O =I ⇥C
<latexit sha1_base64="Rw4UFwukpYQGXy0YZJLiUajRZYc=">AAACDHicbVDLSgNBEJyNrxhfUY9eBoPgKeyKohchkIuejGIekKxhdjKbDJmdWWZ6hbDkFzx71W/wJl79Bz/Bv3A2yUETCxqKqm66u4JYcAOu++XklpZXVtfy64WNza3tneLuXsOoRFNWp0oo3QqIYYJLVgcOgrVizUgUCNYMhtXMbz4ybbiS9zCKmR+RvuQhpwSs9HCDL/E17gCPmMHVbrHklt0J8CLxZqSEZqh1i9+dnqJJxCRQQYxpe24Mfko0cCrYuNBJDIsJHZI+a1sqiV3jp5Orx/jIKj0cKm1LAp6ovydSEhkzigLbGREYmHkvE//1MgWUEmbuAAgv/JTLOAEm6XR/mAgMCmfJ4B7XjIIYWUKo5vYFTAdEEwo2v4LNxptPYpE0TsreWdm9PS1V7mYp5dEBOkTHyEPnqIKuUA3VEUUaPaMX9Oo8OW/Ou/Mxbc05s5l99AfO5w81G5sM</latexit>
Design Sciences applied to Games

It is needed
Games are widely used for better engagement
Games are tough to design
It is possible
AI can be used to model relationships between structure and mind
My work broadly pursues this agenda as applied to narrative
rogelio@cs.utah.edu

University of Utah
03 03 2021
End
Call to Action
Embrace Design
No optimal solutions, only tradeoffs (“it depends”)
Tripartite Model of Games Research

Content Game Cognition
Interface
Seek the invariant relationships
Appendix
Human Cost
Subtitle Games
Telltale Text
Telltale Games
Human Cost
Subtitle Text
Rockstar Games
Graphics at the Expense of Stories
Graphics at the Expense of AI
Graphics at the Expense of AI
Procedural Content Generation
Artificial Intelligence for
Game Content Creation
Games as Conversation
The whole value of a game is in the mental model

of itself it projects into the player’s mind.
Games as Conversation
(Cardona-Rivera and Young, 2014)
Interactive Narratives
Playable systems with significant
narrative elements
Mediate interaction through dramatic framing
Understanding story and game is needed to
advance
Message Objective
We need mathematical study of minds, machines, medium, and message
What is narrative design?

How can we design narratives intentionally?
How can we best support narrative design?
New language: Goals, Feedback,
and Interpretation
Complements: Mechanics,
Dynamics, Aesthetics
Framing: Games as Conversation

Open Question: Theoretical
Orientation for Design?
Preliminary Answer: Situated
Function-Behaviour-Structure
and Interpretation
Rogelio E. Cardona-Rivera, José P. Zagal, and Michael S. Debus; GFI: A Formal Approach to Narrative Design and Game Research. In Proceedings of the 13th International
Conference on Interactive Digital Storytelling (ICIDS 2020), pages 133-148, 2020. (Best Paper Runner Up)
Game Design Gaps 🕳
Language for Narrative Design Needs
Mechanics Dynamics, Aesthetics

Mechanics, Goals
How you can act Why
Mechanics, Technology, Narrative, Aesthetics you would
want to
Mechanics + Systems, Gameplay, Player Experience
Dynamics Feedback
Elements, Behaviours, Experience
How the system reacts How you would know
Aesthetics Interpretati
on
How you feel Why you feel
it
Ultimate Goals In-game objectives or conditions players are
expected to meet to succeed at a game
To Finish
Ultimate Goals In-game objectives or conditions players are
expected to meet to succeed at a game
3 Types
José P. Zagal, Michael S. Debus, and Rogelio E. Cardona-Rivera; On the Ultimate Goals of Games: Winning, Finishing, and Prolonging. In
To Win To Finish To Prolong
José P. Zagal, Michael S. Debus, and Rogelio E. Cardona-Rivera; On the Ultimate Goals of Games: Winning, Finishing, and Prolonging.
In Proceedings of the 13th International Philosophy of Computer Games Conference (PoCG 2019), St. Petersburg,
Russia, 2019.
To Finish
And how do you
accomplish that?
To Finish
Imperative Goals Closer-to-gameplay conditions players are
expected to meet to accomplish the
ultimate
To Finish
Remove
ultimate
To Finish
And how do you

accomplish that?
Remove
ultimate
To Finish
Remove
Reach
Imperative Goals
Imperative Goals
Closer-to-gameplay conditions players are
ultimate
10 Types
Optimize Reach Remove Solve Synchronize
Choose Configure Create Find Obtain
Michael S. Debus, José P. Zagal, and Rogelio E. Cardona-Rivera; A Typology of Imperative Game Goals. Game Studies, 20(3), 2020.
To Finish
Remove
Reach
Ludological Hierarchy
To Finish
Remove
Reach
What does this
mean?
To Finish
To Finish
Save the Princess
To Finish
Save the Princess
Imperative Goals Interpretation of a ludological goal that
depends on the player and the game’s
feedback
This paper.
To Finish
Save the Princess
Rogelio E. Cardona-Rivera, José P. Zagal, Michael S. Debus; Narrative Goals in Games: A Novel Nexus of Story and Gameplay. In Proceedings of the 15th International Conference on the
Foundations of Digital Games (FDG 2020), Malta, 2020.
To Finish
Save the Princess
Remove Defeat Bowser

To Finish
Save the Princess
Reach Destroy Bridge

Parallel Goal Hierarchy
What does this

mean?
To Finish
Save the Princess

To Finish
Save the Princess

Why do you want to
do this?

To Finish
Save the Princess
Why do you want to
do this?

and Interpretation
Rogelio E. Cardona-Rivera and R. Michael Young; Games as Conversation. In Proceedings of the 3rd Workshop on Games and NLP at the 10th AAAI Conference on
Artificial Intelligence and Interactive Digital Entertainment (GAMNLP-14), pages 2-8, Raleigh, NC, USA, 2014.
Zork
The
You are in an open field west of a big white house with a
boarded front door.
A growing body of research in computational
There is a small
models mailbox here.
of interactive narratives has focused on
> analyzing and generating well-structured
narratives
open mailbox
Opening the mailbox reveals:A leaflet.
Our work focuses on developing a model of
> interactive narratives that considers both the
narrative’s structure as well as its effect on the
player
Games The mailbox is relevant.Why

would it be communicated
otherwise?
and Interpretation

and Interpretation

Orientation for Design?
Preliminary Answer: Situated
Function-Behaviour-Structure
Rogelio E. Cardona-Rivera; Foundations of a Computational Science of Game Design: Abstractions and Tradeoffs. In Proceedings of the 16th AAAI Conference on Artificial
Intelligence and Interactive Digital Entertainment (AIIDE 2020), Online, 2020.
Intention requires goal setting
Question Answering
Arthur wants Arthur

disenchante Reason wants
d Excalibur Goa
l
Outcome Arthur
disenchant
sExcalibur Event
Excalibur
Consequence disenchant
ed State
Rogelio E. Cardona-Rivera, Thomas W. Price, David R. Winer, and R. Michael Young; Question Answering in the Context of Stories Generated by Computers. Advances in Cognitive
Systems, 4, pages 227-246, 2016.
Question Answering
Mimesis
Role-play Effect
Fighter Wizar Rogue
d
Ignacio X. Domínguez, Rogelio E. Cardona-Rivera, James Vance, David L. Roberts;

The Mimesis Effect: The Effect of Roles on Player Choice in Interactive Narrative Role-Playing Games . In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI2016), pages 3438-3449, San Jose, CA, USA, 2016.
Question Answering
Role-play

Orientation for Intentional Effect?
Preliminary Answer: The Visual
Narrative Parallel Architecture
The Visual Narrative Engine
The interactivity grammar?
Chris Martens, Rogelio E. Cardona-Rivera, and Neil Cohn; The Visual Narrative Engine: A Computational Model of the Visual Narrative Architecture. In Proceedings of the 8th Annual
Conference on Advances in Cognitive Systems (ACS 2020), Online, 2020.
Parallel Interfacing Narrative Semantics
Dual (Syntax/Semantics) Process Reasoning
Visual Narrative Grammar
Model of the combined VNG+PINS = Visual Narrative Parallel Arch.
Research Question: Can we describe procedures to match posited interfaces?


Scene Graphs
Scene Graphs
A Representation from Computer Vision
Scene Graphs HTNPlans

Hierarchical Task Networks
A Representation from Automated Planning
Representations are Compatible!
Rogelio E. Cardona-Rivera and Boyang Li; PlotShot: Generating Discourse-constrained Stories around Photos. In Proceedings of the 12th AAAI Conference on Artificial Intelligence and
Interactive Digital Entertainment (AIIDE-16), pages 2-8, Burlingame, CA, USA, 2016.
Representations are Compatible!
Scene Graphs HTNPlans

Question Answering
Role-play

Orientation for Intentional Effect?
Preliminary Answer: The Visual
Narrative Parallel Architecture
The interactivity grammar?
Chris Martens, Rogelio E. Cardona-Rivera, and Neil Cohn; The Visual Narrative Engine: A Computational Model of the Visual Narrative Architecture. In Proceedings of the 8th Annual
Conference on Advances in Cognitive Systems (ACS 2020), Online, 2020.
Understanding Players
Goal Elicitation
Adam Amos-Binks and Rogelio E. Cardona-Rivera; Goal Elicitation Planning: Acting to Reveal the Goals of Others. In Proceedings of the 8th Annual Conference on Advances in
Cognitive Systems (ACS 2020), Online, 2020.
Goal Elicitation
Invisible Support
Justus Robertson, Rogelio E. Cardona-Rivera, and R. Michael Young; Invisible Dynamic Game Adjustment in Virtual Reality Games. To appear in
Proceedings of the 3rd IEEE Conference on Artificial Intelligence and Virtual Reality, Online, 2020.
Goal Elicitation
Invisible Support
Orientation for Design Tools?
Preliminary Answers:
Metaphors
Message Objective

Recap

Next Generation Open Questions
VNE (and VNG): Where does interaction fit in?
How does fluency mediate tool use? (VLFI for Games?)

Recap and Wrap: Join me?

Next Generation Open Questions
VNE (and VNG): Where does interaction fit in?
How does fluency mediate tool use? (VLFI for Games?)

Storytelling and Understanding AI: Toward A Critical Technical Practice

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Storytelling and Understanding AI: Toward A Critical Technical Practice

Uploaded by

Copyright:

Available Formats

Storytelling and Understanding AI

Toward A Critical Technical Practice

Rogelio E. Cardona-Rivera, Ph.D. http://rogel.io

Assistant Professor @recardona

qed.cs.utah.edu LABORATORY FOR QUANTITATIVE EXPERIENCE DESIGN

Idea: Computational Narrative Models can help

What is a Computational Model?

The Embodied and Cognitive Age f (x)

The Embodied and Cognitive Age f (x)

The Embodied and Cognitive Age f (x)

The Embodied and Cognitive Age f (x)

The Introspection Age

“May I say that I have not thoroughly enjoyed

“You've shown none of the concern that Captain

An example agenda in the

Unique human capacity to

(Heider and Simmel, 1944)

Narrative framing makes interaction more compelling

Narrative framing makes interaction more compelling

Narrative framing makes interaction more compelling

Narrative framing makes interaction more compelling

Narrative framing makes interaction more compelling

Story generation as a classical planning problem P = hsi , g, Di

Actions encoded as template operators

Planning Domain Definition Language

(:action pick-up :parameters (?agent ?item ?location)

Actions encoded as template operators

Planning Domain Definition Language

(:action pick-up :parameters (?agent ?item ?location)

Actions encoded as template operators

Planning Domain Definition Language

(:action pick-up :parameters (?agent ?item ?location)

Solution to a planning problem is a

Solution to a planning problem is a

Solution to a planning problem is a

Solution to a planning problem is a

(define (domain KNIGHT)

(define (domain KNIGHT) (define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

(define (problem STORY) (:domain

Story Director Player can act as afforded by

Intended Narrative Plan

Pick-up Disenchant Pick-up

On Player Turn: ???

Pick-up Disenchant Pick-up

On Player Turn: ???

Pick-up Disenchant Pick-up

On Player Turn: ???

Pick-up Disenchant Pick-up

Pick-up Disenchant Pick-up

Pick-up Disenchant Pick-up

Wake-up Many many

Pick-up Disenchant Pick-up

Wake-up Many many

Pick-up Disenchant Pick-up

Wake-up Many many

Pick-up Disenchant Pick-up