Episodic Memory and Cognitive Capabilities

Andrew Nuxoll
24 May 2007

Outline
Review and Introduction
Definitions Research Goals TankSoar Domain

Demonstrating Cognitive Capabilities
Action Modeling Virtual Sensors Learning from Past Success and Failure

2

Long Term Memory
Memory

Long Term Memory

Short Term Memory

Declarative Memory

Procedural Memory

Semantic Memory

Episodic Memory

Episodic Memory
Memories of specific events in your past
3

Current Implementation
Encoding Initiation? Storage Retrieval
Output

Long-term Procedural Memory Production Rules

Working Memory

Cue

Input

Retrieved

When the agent takes an action.
4

Current Implementation
Encoding Initiation Content? Storage Retrieval
Output

Long-term Procedural Memory Production Rules

Working Memory

Cue

Input

Retrieved

A portion of working memory is stored in the episode
5

Current Implementation
Encoding Initiation Content Storage Episode Structure? Retrieval
Long-term Procedural Memory Production Rules Episodic Memory

Output

Working Memory

Cue

Episodic Learning
Input Retrieved

Episodes are stored in a separate memory
6

Current Implementation
Encoding Initiation Content Storage Episode Structure Retrieval Initiation/Cue?
Long-term Procedural Memory Production Rules Episodic Memory

Output

Working Memory

Cue

Episodic Learning
Input Retrieved

Cue is placed in an architecture specific buffer.
7

Current Implementation
Encoding Initiation Content Storage Episode Structure Retrieval Initiation/Cue Retrieval
Long-term Procedural Memory Production Rules Episodic Memory

Output

Working Memory

Cue

Episodic Learning
Input Retrieved

The closest partial match is retrieved.
8

Research Goals
Explore the cognitive capabilities granted to an agent with an episodic memory Explore what’s necessary to build an effective episodic memory for a general cognitive architecture
Domain independence Performance

Take inspiration from cognitive psychology

9

Cognitive Capabilities: How do we use Episodic Memory?
Sensing
Detecting Repetition Virtual Sensing Noticing Unusual Input Sense of Identity

Learning
Retroactive Learning Reanalyzing with new Knowledge Explaining Behavior “Boosting” other Learning Mechanisms

Reasoning
Action Modeling Recording Previous Successes/Failures Modeling the Environment Managing Long Term Goals

10

TankSoar Domain
Tanks in a maze Sub-goals
Shoot other tanks Don’t get shot Don’t run out of
Energy Missiles

Multiple sensors and actions
11

Action Modeling

Cognitive Capability: Action Modeling
Action Modeling
Definition: Learning the immediate effect of an action Analog: Case-Based Reasoning

Task: Conserve Energy
Selecting proper radar setting to minimize energy consumption

13

Agent Implementation
Radar Setting: 1 Radar Setting: 2
X=11 Y=3 Dir=south Agent’s State


Radar Setting: max
e at ry re C mo e e M Cu

Evaluation Result: Radar Setting: 5
Agent Confirms Location and Direction Match

Episodic Retrieval X=11 Y=3 Dir=south X=11 Y=3 Dir=south

14

Initial Performance
Radar Tank Performance
1 0.9 0.8 Fraction Correct 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1 101 201 301 401 501 601 701 801 901 Radar Settings (time)

Random

15

Nuggets
Episodic memory is an effective medium for action modeling

Coal
Requires agent have knowledge of whether a retrieved episode is useful

16

Virtual Sensors

Cognitive Capability: Virtual Sensors
Virtual Sensors
Definition: Retrieving past sensing that is relevant to the current task

?

Task: Locate the Battery
Using episodic memories to construct a path

18

Agent Implementation
Move North Move West Move East
Agent’s State
e at r y e Cr mo e e M Cu

Evaluation Result: Move North

Episodic Retrieval

19

Performance
Energy Search Time
1000

Number of Moves

100 Random Ep. Mem. 10

1 1 2 3 4 5 6 7 8 9 10 11 12 Searches

20

Paths to Battery

21

Nuggets
Episodic memory can be used as a virtual sensor

Coal
Limited investigation of integration of episodic and semantic memory

22

Learning from Past Success and Failure

Cognitive Capability: Learning from Past Success and Failure
Learning from Past Success and Failure
Definition: Using past performance to guide future behavior More emphasis on long term

Task: Combat
Using episodic memory to determine best tactics in the “attack” subgoal
24

Agent Implementation
Move Forward Move Forward + Fire Missile


Turn Right
Agent’s State
te e rea y Cu C r o em M

Evaluation Result: Move Forward + Fire = 2 points
Compare Memories In the Sequence

Episodic Retrieval

Retrieve the Next Memory in Sequence

Retrieve the Next Memory in Sequence

25

Performance
20 10 0 Margin of Victory 1 -10 -20 -30 -40 -50 -60 Successive Games 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161

26

Tactics Learned
Fight or flight Back away and shoot Dodging

27

Without Heuristic Cue
Performance without Heuristic Cue Selection
20 10
Average Margin of Victory

0 -10 -20 -30 -40 -50 -60 Successive Games 1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161

28

Without Discount Factor
No Discount Factor
20 10
Average Margin of Victory

0 -10 -20 -30 -40 -50 -60 Successive Games 1 12 23 34 45 56 67 78 89 100 111 122 133 144 155 166 177

29

Nuggets
Episodic memory can be effective at learning long term tactics

Coal
Requires the use of a discount factor Requires heuristic cue selection

30

Summary
Episodic memory enables multiple cognitive capabilities including:
Action modeling Virtual sensing Learning from past success and failure

31