General study hints

• Notice slides that re-occur throughout the course

• “Last week:”, “This week”, & “Learning Outcomes” slides are important topics
• Listen to the lectures to see what is discussed and what is not!
– Some material on the slides is there for context and to give interested students
something to follow up on, either now or in a future course/job - this material is
not going to be on the quiz.
• Quiz 1 was science theory & experiments (“design & the brain”) for Wave 1
HCI
– Human Information Processing of sense data
– lab science focused, ask about theory and experiments.
– Much of this is factual knowledge
• Design implications are
– Our intuitions about how our brains work are wrong
– Design should be consistent with science : hence “design & the brain”
 General study hints 2

• First part asks for factual knowledge on memory, mental models, GOMS,
intelligence.
– Multiple choice, fill-in-the-blank
– ~ 20% draw from Designing with Mind in Mind on issues discussed in class
– ~ 10% from Noba on early material
• Second part is on design approaches for Wave 2 & 3 HCI (“affordances &
environment”)
– Smart-seeing & Projecting: how thinking is supported by designed representations
– Close-coupling: how thinking can be shaped by interaction with active systems
– Socially-distributed cognition: how thinking can be done in partnership with people
(and AI agents) using technology to communicate
• Part 2 has fewer facts, more thinking about systems of people and technology
– Some longer written questions — 2 to 3 sentences max
– Look for understanding of key concepts rather than just recall
Early slides that keep coming back (to haunt you)
 How would you approach this after 201?

Schön:Reflective practice skill in “language”

add new knowledge

Arthur: “design
Norman: language” of
Craft Cognitive Engineering phenomena and how
for HCI they are controlled

use design science

new vocabulary from

Simon: “science of the artificial” sciences

!5
 Schön Reflective Practice

THE REFLECTIVE LEARNING CYCLE

• Key aspects are to reflect on what you are Evaluate decision
Schön, D.A. (1984), (1990)

doing and how you can do it better Other

• Formulate a question that can be PRACTICE
triggers

researched in the literature, in an

experiment, interview, questionnaire, or a
Formulate a
field study Integrate decision
into practice question

• Seek information, analyze it and

integrated it in your next project.
Seek
information
Make a
decision

Analyze/
interpret/
reflect
 Technology meets Psychology

• In “The Nature of Technology…” Arthur says tech designers

should build a “language” of and how they can be controlled
by “method technologies”
• Can we build “method languages” by which psychological
phenomena can be enabled, supported and controlled?
– For HIP psych (Wave 1 HCI) cognitive architecture, SS&P
– For social science (Wave 2 HCI) socially-distributed cognition
– For user experience (Wave 3 HCI) Close-coupling
– For distributed cognition (Multiwave HCI) combine the above

!7
 Marr’s multiple perspectives as a guide

• Marr’s “Vision” describes 3 perspectives

• We understand Mechanisms through study of cognitive
architecture— neuroscience, lab studies of human
performance with simple displays (e.g. attention studies
by Treisman, Pylyshyn, etc.)
• We understand Algorithms through study of human
performance in more complex tasks and environments-
(e.g. GOMS and Keystroke analysis)
• Algorithms combine multiple mechanisms and so must
be consistent with them
• We understand Requirements (Marr’s Computational
theory) by studying environments and human behaviour
(social science methods, field studies)
Lecture 7: Mental representations,
learning & mental models
 Design for cognition (thinking)

• Design for diverse user population

– Information presentation for understanding
– Interaction design for better task performance
• Design of groupware for collaboration and coordination

Diversity of users ranging from novice expert

collaborating on tasks that require analysis,
judgment, and coordination
Bill Buxton/Dave Kasik
 Noba "Memory" and "Intelligence"

• Learning Objectives
– Define and note differences between the following forms of
memory: working memory, episodic memory, semantic
memory, collective memory.
– Describe the three stages in the process of learning and
remembering.
– Describe strategies that can be used to enhance the original
learning or encoding of information.
– Describe strategies that can improve the process of retrieval.
– Describe why the classic mnemonic device of the method of
loci works so well.
 Representations from perception to action

• Sensory memory 缓冲:图像(视觉)，回声(听觉)，触觉(触觉)

– Buffers: iconic (visual), echoic (auditory), haptic (touch)

– Limited processing “logic of perception”, sensory learning
Perception
– Passes into short-term memory by attention (filtering)

• Working memory is short-term

– Faster access, rapid decay
– Limited capacity (“scratch-pad”): “Seven plus or minus two”
–
–
We can chunk information to extend this (telephone #’s)
“Flush” when finished with a task
Cognition
– Or, move into long-term via rehearsal
• Long-term memory is slower, larger
– Virtually unlimited capacity (we don’t know how much)
– Slow access, little decay
– Complicated operation that depends on recent access
• Motor memory is special case Motor
– motor programs/perception-action loops

Information processing view

 Working Memory Stores & processes
(Badderly)
• Central Executive: coordinates the process. CE controls:
– Visual-spatial sketchpad: working memory for events in visual (and
visuomotor) space Visual-spatial sketchpad: working memory for events in visual (and visuomotor) space

Articulatory (phonological) loop: working memory for sounds, especially language

– Articulatory (phonological) loop: working memory for sounds,
especially language
– Also involved in attention to task, strategy etc.
• Working memory requires effort, it may be a process rather than a
separate memory representation
– CE operations could take place on LTM stores

V-S
A/P Loop Central Sketchpad
Executive

!13
 “Chunking” can extend A/P Loop STM

• “The magical number seven, plus or minus two: some

limits on our capacity for processing information”
George A. Miller, Psych Review, 1956
• Later determined that AP Loop has storage of 4+-1
“chunks”, not items-- Miller’s subjects were “chunking”
sequences of numbers
– Example: area code 604 is a “chunk” because it occurs
so often. It takes only one place in the AP Loop.
– Are there other “chunks” you can think of?

!14
 Conclusion from Chase & Simon (1973)

• Chess masters only expert with real chess positions.

They do not have better memory in general

• Expertise allows chunking of salient information to

promote memory of good moves
专业知识允许对重要信息进⾏分块，以提⾼对好动作的记忆

• Experts organize knowledge differently – reflects a

deep understanding.
 Experts represent a problem at a deeper
level

• Experts see structural similarities-- they think only

about aspects of the problem that are important to
its solution, so their cognitive load is low
• Novices see surface similarity-- they think about
unimportant aspects of the problem as well as
important ones, so their cognitive load is high
• When you acquire expertise, you learn how to
lower your cognitive load so that the problem
becomes easier

(Chi, Glaser, and Farr, 1988)

 10,000 hour rule

• 10K hrs of deliberate practice to attain international level

• Deliberate practice: highly motivated w careful self-
monitoring
• Master chess players play 10,000 – 20,000 hours
Understanding the world: mental
representations and procedures
 Keystroke Level Model of task performance

• Card, et al., 1983.

– observable events
– a single stream of sequential
operators.
• provide crude task-execution
times.
• Limitations:
– Procedural (vs. tacit,
declarative) knowledge
– Routine cognitive skills
– Designer must know top-level
tasks and user goals
 Common kinds of mental models

• State transition model

– Changes in state need to be “visible”
– Telephone example
• Object-action models
– Users think in terms of concrete or abstract objects
– The system supports action on the objects
– Unix mv example
• Mapping models
– Users learn a sequence of actions to accomplish tasks
– Hand-held calculator maps “math” to key presses
• Analogies
– A new system (closely) resembles an old system
CPM-GOMS PERT example
How can
mental
models be
used in
design?
 Thinking about expertise and intelligence

• We often think of intelligence as:

– The ability to solve problems
– One thing “high IQ”
– Something that we can measure objectively
– Something that is independent of the situation &
environment
• What happens when problem solving ability changes in
different situations? Can intelligence be a function of the
environment?
 Early Intelligence was general processing
capability
• Focus on General Intelligence or “G” — ability to reason
about a variety of things
– early Stanford-Binet test was mostly conceptual-
linguistic
– Later tests included other kinds of questions to broaden
the scope of intelligence
– Modern intelligence tests ask many different kinds of
questions, and combine the results for a single IQ score
 Intelligence — processing capability

• IQ tests were/are much criticized, variants developed

– Cultural, environmental differences may change results
– Stanford-Wechler test measures multiple kinds of
intelligence (e.g. “visual intelligence”)
– In Human Cognitive Abilities, John Carroll used factor
analysis
• Fluid vs crystallized intelligence
• General memory & learning, retrieval
• “Speediness” measures
• Sensory intelligence (vision, hearing)
 Types of Intelligence

Carroll’s Model

▪ Fluid
▪ Crystallized
 Gardner's Theory of Multiple Intelligences

www.literacynet.org/mi/assessment/findyourstrengths.html

Logic-math Visual-spatial Music-rhythm Verbal-linguistic

Bodily-kinesthetic Interpersonal Intrapersonal Naturalistic

 Ecological view of expertise (intelligence)

Perception

Affordances Cognition
Actions

Motor
Physical affordances
Perceived affordances
Lecture 9: Design for intelligence—
Augmenting cognition with external representations
Accidental augmentation
 Implications for design— Can computers
make us smart?

• Traditional HCI focuses on thinking-

in-the-head , but interfaces have
become visually complex and have
rich (e.g. touchscreen) interaction

• Can an understanding of how

people think with images, symbols,
data visualization, games etc.
expand our cognitive capabilities?

• Should be be teaching kids how to

do think with their eyes and hands?
 Is the bird intelligent?
How about the pilot?

The senses considered as perceptual

systems (1966)
J.J. Gibson
 Ecological view of education

• Bloom’s taxonomy
included cognitive,
Cognitive Affective Psychomotor
motor and affective
elements
evaluate

• Generally we only synthesize internalize naturalization

hear about the
cognitive aspects analyze organize articulation

apply value precision

• Why is that?!?
understand respond manipulate

recall data receive imitate

Lecture 10: Intro to distributed cognition:
Smart-seeing and Projection
 Smart Seeing and Projecting

• Smart Seeing & Projecting argues that thinking can be

distributed between internal process and external
representations (e.g. symbols, sketches) or tools in the
environment.
• A person working with a tool can be a distributed cognitive
system
 Coordinating mental representations &
visual objects
• Why do chess players
prefer to play with a
chess board in front of
them?
• Remember the
chessboard memory
studies studies
• What happens after
they see the patterns
on the board?

https://www.nytimes.com/2019/03/16/opinion/sunday/chess-
champion-8-year-old-homeless-refugee-.html
Projec'ng posi'ons
 Design implications of Smart Seeing & Projecting

• Can an understanding of how people

think using images, symbols, data
visualization etc. inform interface
design?

• Will future interfaces use these

methods to augment human
intelligence, thus making us
smart(er)?

• Should be be teaching kids how to do

think with their eyes and hands so
they can use these new tools?
James Clerk Maxwell (1831-1879)

• Black was
drawn by
Maxwell
• Blue describes
what Maxwell
knew, linked to
drawn
Working on the paper: Feynman
diagrams
Peter Cheng’s visual languages
 Producing images to think with
Interac(ng with representa'ons makes us more powerful
thinkers.
 Smart-Seeing & projecting with no time
constraints

Repeat in sequence:

1. Evaluate the board

with smart-seeing
2. Project possible
moves & responses
3. Select the best move
4. Make the move
5. Wait for response

https://www.nytimes.com/2019/03/16/opinion/sunday/chess-
champion-8-year-old-homeless-refugee-.html
 Advanced form

Much more complex

reasoning, but still iterative
• Sketches capture ideas, Maryam Mirzakhani
concepts, frameworks & 1977-2017

options better than symbols

• Words and symbols are
used where appropriate
• Process is also iterative,
with no time pressure

Maryam Mirzakhani 1977-2017

 What types of intelligence are used?

Carroll’s Model

▪ Fluid
▪ Crystallized
Colin Ware’s view
 Brett Victor’s Kill Math Project

• http://worrydream.com/KillMath/
• https://vimeo.com/worrydream/videos
 Brett Victor’s Kill Math Project

• http://worrydream.com/KillMath/
• https://vimeo.com/worrydream/videos
Algebra for kindergarteners
Close coupling
 How is the Gannet’s task harder than the
mathematician’s?

Gibson says senses should be thought of

as perceptual systems that combine
sense information, environment, and
actions that can be taken
 The Gannet’s task is dynamic

• In the case of the Gannet,

apparent target position
changes automatically in
response to wind, waves,
and motion of the fish
• The Gannet has to respond
to these in real-time in order
to succeed
• Easy in slow or static
environments— calm
waters & steady winds
• Harder in more dynamic
environments— heavy
wind, more waves
 Learning outcomes for this week: Close-
coupling and multi-agent cognition

• Understand Close-coupling: how a thinker uses embodied

intelligence to coordinate thinking with speed of information
– Report changes in the world— e.g. air traffic control
– Computation based on changes in the world — e.g.
Waze/Google maps route advice
– Software generated situations (e.g. games)
• Think about multi-agent cognitive systems
– How people coordinate with each other using language
– How to design environments to support collaboration
– How to extend to computational agents
 What types of intelligence are used?

Carroll’s Model

▪ Fluid
▪ Crystallized
 Computer games and fluid intelligence

Abstract
Hundreds of millions of people play intellectually-demanding video games every day. What does individual
performance on these games tell us about cognition? Here, we describe two studies that examine the potential
link between intelligence and performance in one of the most popular video games genres in the world
(Multiplayer Online Battle Arenas: MOBAs). In the first study, we show that performance in the popular MOBA
League of Legends’ correlates with fluid intelligence as measured under controlled laboratory conditions. In the
second study, we also show that the age profile of performance in the two most widely-played MOBAs (League
of Legends and DOTA II) matches that of raw fluid intelligence. We discuss and extend previous videogame
literature on intelligence and videogames and suggest that commercial video games can be useful as 'proxy'
tests of cognitive performance at a global population level.
 Closely coupled system process and
structure migrates wherever costs are lowest

Wave 1 HCI view (Norman)

• “Perceived affordances” in interface

lets user work with simpler mental model

Internal

Processes External Processes Dcog view (Kirsh)
• Users adapt to cost structure
• Epistemic (computational)
• Pragmatic (physical) costs
• Process is not just representational
• User actions create new structures
Human equivalents
 J.C.R. Licklider "man-computer symbiosis”:
computers as "cognitive partners"(1960)


“The hope is that, in not too many years, human brains and
computing machines will be coupled together very tightly and that
the resulting partnership will think as no human brain has ever
thought and process data in a way not approached by the
information-handling machines we know today.”
Embodied intelligence
 Method tech for Close Coupled systems?

• Tech designers build a

“language” of
phenomena that they
use and how it can be
controlled by “method
technologies” 视觉眼肌运动的

• What are example • Visuomotor routines

phenomena of Close • Expert performance
Coupling?
• What methods could
be developed to • Cognitive task analysis
support them? • Game design
Socially-distributed cognition
(multi-agent systems)
 Socially distributed cognition

• Roles or parts to play

• Timing: When to do what – rules and procedures to follow
• Pass information: text, symbols & sketches, voice, video
• Read meters, transform information
• Group structure is often emergent
 What types of intelligence are used?

Carroll’s Model

▪ Fluid
▪ Crystallized
 Collaborative cognition

• “In ‘The Knowledge Illusion,’ the cognitive scientists

Steven Sloman and Philip Fernbach hammer another nail
into the coffin of the rational individual…Sloman and
Fernbach take this argument further, positing that not just
rationality but the very idea of individual thinking is a myth.
Humans rarely think for themselves. Rather, we think in
groups. Just as it takes a tribe to raise a child, it also
takes a tribe to invent a tool, solve a conflict or cure a
disease. No individual knows everything it takes to build a
cathedral, an atom bomb or an aircraft. What gave Homo
sapiens an edge over all other animals and turned us into
the masters of the planet was not our individual rationality,
but our unparalleled ability to think together in large
groups.”

https://www.nytimes.com/2017/04/18/books/review/knowledge-illusion-steven-sloman-philip-fernbach.html
Docking this ship needs knowledge & intelligence

• Body Level One

– Body Level Two
• Body Level Three
– Body Level Four
» Body Level Five
 “Cognition in the Wild” Hutchins 1995

• Cognitive Ethnography methodology to study how

– Many people must coordinate actions, but none may
understand the complete process
– Task can be reliably accomplished even when no one present
knows how to to it
• Cognitive system includes:
– “Scripts”-- sequences of tasks in which individuals with
specific roles are trained
– Artifacts that support the task (technologically distributed
cognition)
– Communication protocols and channels for coordination
 Piloting & navigation as a cognitive system

• Computational subunits (person/task/artifact)…

• Encapsulate information needed for operation
• Be at the correct location (on the ship)
• Act at the correct time so as to coordinate with other actors
and events on the ship and in the environment
• Communicate information when needed
• System as a whole
• Can correct errors
• Supports dynamic reconfiguration
• Can be analyzed at multiple levels and loci

(Kirsh)
 Where’s the mind?

Hutchins says:
• When looking at how a ship gets docked it is the
system that has the mind – it has mental properties
• Mind is the thing with the knowledge – which in the
ship is distributed over people, procedures & things.
Intelligence implies cognition and is displayed in how
robust, resilient, adaptable the system is in solving a
hard problem
• “Cognition in the Wild” book compares Micronesian
canoe and navy ship navigation
Starbucks

(Kirsh)
 Starbucks Revolutionary Technology

• Changes cognitive
efficiency of whole
system
• Minimizes costs in
most areas

Form on cup

Technology of coordination
(Kirsh)
 Method tech for multi-agent systems?

• Tech designers build a

“language” of
phenomena that they
use and how it can be
controlled by “method
technologies”
• What are example
phenomena of Socially- • Mental models of agents
distributed cognition? • Communication methods
• Interface “signifiers”
• What methods could
be developed to Collaboration design
support them? (CSCW) …
 Designing for coordination: social signifiers

People need some way of understanding the product or service-some sign of what
it is for, what is happening, and what the alternative actions are…Forget
affordances: What people need, and what design must provide, are signifiers.
Because most actions we do are social, the most important class of these are social
signifiers…. Social signifiers replace affordances, for they are broader and richer,
allowing for accidental signifiers as well as deliberate ones, and even for items that
signify through their absence, as the lack of crowds on a train

Interactions Dec-Jan 2008

Engelbart knew it all along…
Networked Improvement Community (NIC)
“Augmented” Community, using best-practice
Collective-IQ Capability Engelbart slides
Sharing risk, cost, knowledge, experience

Networked Improvement Community (NIC)

Dynamic Knowledge Repository

Org 1

Recorded Knowledge Intelligence 


Org 2 Dialog Products Collections

• Investigate & collect intelligence 


...

• Share experience of using leading-edge,

collective-IQ practices. 

• Objective: provide best understanding of
Org N the “Co-Evolution Frontier” for
augmenting this capability.

!79
 Distributed Cognitive Systems

• Human information processing (psych) model: thinking in the

head
• Offload cognition to representations you can manipulate, e.g.
Sketching, math on paper etc. (Smart-seeing & projecting)
• "Close-couple” skilled human & active/responsive artifact
• Musical instruments
• Technological: Games and “game-ified” interfaces
• Distribute problem across people with action scripts,
discussion, & action coordination
• Communities of interest, e.g. Linked-In
• Broader society, e.g. “See, click, fix”, Placespeak
• Structured crowdsourcing methods
 Borg as an information system

Borg have coordinated thought – they distribute their problems for parallel,
distributed processing
Culture uploads ‘programs’ to us
 Social science & HCI

• Ethically-aligned design uses social science methods to guide

design for the benefit of society