Touch Research

A project for Communication Design M1

HTWG Constance
Summer Term 2008

Part 3: Reading a Research Paper

How Bodies Matter: Five Themes for Interaction Design, DIS 2006,
June 2628, 2006
With the graphical user interface (GUI) so popular these days ...
... this reduced body symbolizes how we use our computers. With

keyboard (fingers),

mouse (fingers),

monitor (eyes), and

speakers (ears).
Are richer interaction paradigms possible?
The tangible user interface (TUI)!

A tangible user interface (TUI) is a user interface in which a

person interacts with digital information through the physical
environment. The initial name was graspable user interface, which
no longer is used.

Image: Nintendo Wii

Bodies matter!

Consider riding a bicycle: one is simultaneously navigating,

balancing, steering, and pedaling; yet it is not possible for
bicyclists to articulate all of the nuances of an activity that
they successfully perform.

Perhaps the most remarkable aspect of this is that riding a

bicycle is just one of thousands of activities that our bodies
can do.

Our physical bodies play a central role in shaping human

experience in the world, understanding of the world, and
interactions in the world.

This paper draws on theories of embodiment from psychology,

sociology, and philosophy synthesizing five themes we believe
are particularly salient for interaction design.
Contrast the richness, subtlety, and coordination of tasks at
several levels of concern that bicycling offers with the
graphical user interface that we use today.
Douglas C. Engelbart, oN-Line System (NLS) and computer mouse,
1960s

A revolutionary computer collaboration system designed by Douglas

Engelbart and the researchers at the Augmentation Research Center
(ARC) at the Stanford Research Institute (SRI).

The first system to employ the practical use of hypertext links,

the mouse (co-invented by Engelbart and colleague Bill English),
raster-scan video monitors, information organized by relevance,
screen windowing, presentation programs, and other modern
computing concepts.

Features: The mouse, 2-dimensional display editing, in-file

object addressing, linking, hypermedia, outline processing,
flexible view control, multiple windows, cross-file editing,
integrated hypermedia email, hypermedia publishing, document
version control, shared-screen teleconferencing, computer-aided
meetings, formatting directives, context-sensitive help,
distributed client-server architecture, uniform command syntax,
universal "user interface" front-end module, multi-tool
integration, grammar-driven command language interpreter,
protocols for virtual terminals, remote procedure call protocols,
compilable "Command Meta Language
Altair 8800, 1975

The MITS Altair 8800 was a microcomputer design from 1975 based
on the Intel 8080 CPU and sold by mail order through
advertisements in Popular Electronics, Radio-Electronics and
other hobbyist magazines.

The Altair is widely recognized as the spark that led to the

microcomputer revolution of the next few years: The computer bus
designed for the Altair was to become a de facto standard in the
form of the S-100 bus, and the first programming language for the
machine was Microsoft's founding product, Altair BASIC.

Programming the Altair was an extremely tedious process. The user

toggled the switches to positions corresponding to an 8080
microprocessor instruction or opcode in binary, then used an
'enter' switch to load the code into the machine's memory, and
then repeated this step until all the opcodes of a presumably
complete and correct program were in place.

When the machine first shipped the switches and lights were the
only interface, and all one could do with the machine was make
programs to make the lights blink.
Xerox Star, 1981

The Star workstation, officially known as the Xerox 8010

Information System, was introduced by Xerox Corporation in 1981.
It was the first commercial system to incorporate various
technologies that today have become commonplace in personal
computers, including a bitmapped display, a window-based
graphical user interface, icons, folders, mouse, Ethernet
networking, file servers, print servers and e-mail.

The key philosophy of the user interface was to mimic the office
paradigm as much as possible in order to make it intuitive for
users. The concept of WYSIWYG was considered paramount. Text
would be displayed as black on a white background, just like
paper, and the printer would replicate the screen using
InterPress, a page description language developed at PARC.

The user would see a desktop that contained documents and

folders, with different icons representing different types of
documents. Clicking any icon would open a window. Users would not
start programs first (e.g. a text editor, graphics program or
spreadsheet software), they would simply open the file and the
appropriate application would appear.
Apple iMac, 2008 edition

Members of the Apple Lisa engineering team saw Star at its

introduction at the National Computer Conference (NCC '81) and
returned to Cupertino where they converted their desktop manager
to an icon-based interface modeled on the Star. Among the
developers of the Gypsy editor, Larry Tesler left Xerox to join
Apple in 1980 and Charles Simonyi left to join Microsoft in 1981
(whereupon Bill Gates spent $100,000 on a Xerox Star and laser
printer),[9] and several other defectors from PARC followed
Simonyi to Microsoft in 1983.

Some people feel that Apple, Microsoft, and others plagiarized

the GUI and other innovations from the Xerox Star, and believe
that Xerox didn't properly protect its intellectual property.
Dan Saffer, reviewing the paper:

With the current keyboard-mouse-monitor set-up, we do every

task, no matter if it is writing a paper or editing a movie or
even playing a game, all the same way. Pointing, clicking,
dragging and dropping, etc. The work has become homogenized.

http://www.odannyboy.com/blog/new_archives/2008/05/
review_five_the.html
Five themes for interaction design:

1. Thinking through doing

2. Performance
3. Visibility
4. Risk
5. Thickness of practice
1. Thinking through doing

describes how thought (mind) and action (body) are deeply

integrated and how they co-produce learning and reasoning.
Individual corporeality.

Unlike theories of information processing and human cognition

that focus primarily on thought as something that only happens in
the head, theories and research of embodied cognition regard
bodily activity as being essential to understanding human
cognition.

Dan Saffer: There are a lot of skills you simply cannot learn by
reading or listening alone. You have to try them out. Gestures
aren't just for embellishment to communication, they can also be
an aid to learning and understanding. Manipulation of items
allows for greater understanding of the item. Artifacts have
their own characteristics, and their "backtalk" uncovers problems
or can suggest new designs.
1.1 Learning Through Doing

Montessori toys employ bodily engagement with physical objects to

facilitate active learning.

Being able to move around in the world and interact with pieces
of the world enables learning in ways that reading books and
listening to words do not.
Having the hands stuck on a keyboard are likely to hinder the
thinking and communication.

1.2 The Role of Gesture

From studies of gesturing in face-to-face interactions, we know

that people use gesture to conceptually plan speech production
and to communicate thoughts that are not easily verbalized.

Gesturing has been shown to lighten cognitive load for both

adults and children.

Less constraining interaction styles are likely to help users

think and communicate.
1.3 Epistemic Action

Distinguishing pragmatic action manipulating artifacts to

directly accomplish a task from epistemic action manipulating
artifacts to better understand the tasks context.

Puzzle players manipulate pieces to understand how different

options would work.
1.4 Thinking through Prototyping

Reflective practice, the framing and evaluation of a design

challenge by working it through, rather than just thinking it
through, points out that physical action and cognition are
interconnected.

Successful product designs result from a series of conversations

with materials.

The epistemic production of concrete prototypes provides the

crucial element of surprise, unexpected realizations that the
designer could not have arrived at without producing a concrete
manifestation of her ideas.
1.5 On Representation

The representation of a task can radically affect our reasoning

abilities and performance.

Tangibility offers both direct familiarity and a set of common

metaphors to leverage in interaction.

The most common stated purpose of tangibility is that these

interfaces provide natural mappings and leverage our
familiarity with the real world, e.g., virtual objects are
positioned in virtual space by moving physical handles in
physical space.
2. Performance

describes the rich actions our bodies are capable of, and how
physical action can be both faster and more nuanced than symbolic
cognition . Individual corporeality.

One of the most powerful human capabilities relevant to designers

is the intimate incorporation of an artifact into bodily practice
to the point where people perceive that artifact as an extension
of themselves; they act through it rather than on it.

While much of the recent TUI literature has focused on walk up

and use scenarios which require a low use threshold, this
section describes how designing for skilled bodies can yield
interfaces for expert performance. Physical interfaces with
dedicated (i.e., spatially multiplexed) controls and dedicated
actions can leverage this skill to improve interaction speed and
reliability.
Dan Saffer: We should design products for expert users, able to
use their hands and motor memory to perform action-centered
skills. Thinking can be too slow; experiential cognition (learned
skillful behavior like driving a car) can be more rapid and
powerful than reflective cognition.

2.1 Action-centered Skills

The tacit knowledge that many physical situations afford plays an

important role in expert behavior.

We draw attention to the importance of tacit knowledge because

computerization can, often accidentally, inhibit it.
2.2 Motor Memory

We are able to sense, store and recall our own muscular effort,
body position and movement to build skill.

It is this motor, or kinesthetic, memory that is involved in

knowing how to ride a bicycle, how to swim, how to improvise on
the piano.

Traditional GUI interfaces employ the same bodily actions for a

wide variety of tasks - this universality is both a strength and
a weakness.

For any given application, kinesthetic memory can only be

leveraged to a limited extent since the underlying actions are
the same across applications.
2.3 Reflective Reasoning

Beyond reliability and robustness of kinesthetic recall, speed of

execution also favors bodily skill for a class of interactive
systems that require tight integration of a human performer in
the loop.

Many daily actions such as driving a car or motorcycle, operating

power tools, or engaging in athletic activities require complex
yet rapid bodily responses for which planning through explicit
cognition is simply too slow.

Tangible interfaces that engage the body can leverage body-

centric experiential cognition. To date, computer game
controllers have been the most commercially successful example of
such interfaces.
2.4 Hands

Simultaneously a means for complex expression and sensation:

Hands allow for complicated movement but their skin also has the
highest tactile acuity of our extremities.

Many of the complex motions that we perform are bi-manual and

asymmetric.
3. Visibility

describes the role of artifacts in collaboration and cooperation.

Social affordance.

The extent to which the activities of a practice are made visible

to colleagues and onlookers through the performance of the
activity.

Dan Saffer: Through the performance of an activity, that

activity can be made visible to others easily, so that
collaboration and situated learning can occur spontaneously.
Visibility facilities

3.1 Coordination

The visibility provided through collocated practice with task-

specific artifacts is also successful in supporting synchronous
collaboration, and can be especially useful in mission-critical
systems.
3.2 Situated Learning

The invisibility of work practice that the GUI has brought about
inhibits peripheral participation.

A child watching to see what an adult is doing in front of a

laptop, and then copying those motions is not learning anything.

With the graphical interface, there is no mechanism to be aware

of the practices of experts; it all looks the same.
Thats what

3.3 Live Performance

is about

The value we place in visibility of creative production is

exemplified by live musical performance.

While the music itself is more intricate and polished in studio

recordings, audiences still pack concert venues because live
perform-ances permit listeners to witness the act of performance
as well as co-produce the event (musician and audience respond to
each other through mutual feedback).
4. Risk

explores how the uncertainty and risk of physical co-presence

shapes interpersonal and human-computer interactions. Social
affordance.

Dan Saffer: Most products are designed to decrease risk, but

retaining some risk can be beneficial. With risk comes trust,
responsibility, and attention.
4.1 Physical Action is Characterized by Risk

Risk is having to choose an action which cannot be undone while

the consequences of the action are not fully knowable ahead of
time.

One cannot undo a social faux pas in face to face interactions;

technology mitigates against this risk: one can delete sentences
before sending them to friends over IM or email.
4.2 Trust and Commitment

Though risk can make people feel more anxious about interactions
with others, it can also engender the kind of trust necessary for
successful distance collaborations.

Situations that involve more risk can also stimulate more

committed involvement by participants of the interaction.

Painting in watercolor requires more commitment to each stroke

than working in Adobe Photoshop.
4.3 Personal Responsibility

There are situations where the decision-makers should not be

subject to the overwhelming repercussions of their decisions,
e.g., natural disaster response planning.
4.4 Attention

Situations of higher risk cause people to feel more emotion-ally

negative and, therefore, more focused, paying closer attention to
detail, while situations of low risk allow people to feel more
emotionally positive, relaxed, curious, and creative.
5. Thickness of practice

suggests that because the pursuit of digital verisimilitude is

more difficult than it might seem, embodied interaction is a more
prudent path.

Dan Saffer: Because there is so much benefit to the real world,

we should be careful with replacing physical artifacts with
digital ones. The best case scenario is to augment the physical
world with digital behaviors, and thus "admitting the
improvisations of practice that the physical world offers."
5.1 Final Scratch

From a design perspective, solutions that carefully integrate the

physical and digital worlds leaving the physical world alone to
the extent possible are likely to be more successful by
admitting the improvisations of practice that the physical world
offers.

Clearly, the digital world can provide advantages. To temper

that, we argue that because there is so much benefit in the
physical world, we should take great care before unreflectively
replacing it.
5.2 Embodied Virtuality Rather Than Virtual Reality.

Designing interactions that are the real world instead of ones

that simulate or replicate it hedges against simulacra that have
neglected an important practice.
In this paper we developed our view of the affordances of
physicality and concreteness for the design of interactive
systems.

We believe the five themes presented in this paper will be of

value both generatively helping designers come up with new
solutions and for evaluation providing a rich set of axes for
analyzing the benefits of systems.

Scott R. Klemmer, Bjrn Hartmann, Leila Takayama

Credits
Via Creative Commons on flickr:

Javier Martnez www.flickr.com/hyoga/1165367241/

raysto www.flickr.com/raysto/252630833/
duncan c www.flickr.com/duncan/92623025/
Kit Cowan www.flickr.com/kitcowan/712113879/
Chel's piccies www.flickr.com/mich_chel/1858616605/
Stefan Sonntag www.flickr.com/zerega/1029076197/
foreversouls/ tree & j hensdill www.flickr.com/foreversouls/
2083165757/
Derrick Mealiffe www.flickr.com/dmealiffe/171720479/
Thomas Hawk www.flickr.com/thomashawk/2492298772/
Franois @ Edito.qc.ca www.flickr.com/francois/1161267539/
Philip Milne www.flickr.com/pamilne/1392285543/
Tom Adriaenssen www.flickr.com/inferis/266391949/
missionbycicles www.flickr.com/mission_bicycles/2108803461/
Chris Gansen www.flickr.com/daychokesnight/2041048614/
Connie Arida/ Connie Sec www.flickr.com/conniesec/2211700732/
Bruce Beh www.flickr.com/brucebeh/120075758/
all in green/ Shana www.flickr.com/isoldesmom/446843362/
Via Creative Commons on flickr:

SOCIALisBETTER www.flickr.com/27620885@N02/2597329151/
"Cowboy" Ben Alman www.flickr.com/rj3/2562634563/
Loran Tatooine www.flickr.com/noloran/641931694/
Luca Casamassima www.flickr.com/tyler89/2364065361/
Axel Pfaender www.flickr.com/axor/2295802927/
rickz www.flickr.com/rickz/3071093/
Jenny Downing www.flickr.com/jenny-pics/2583485800/
Daniel Y. Go www.flickr.com/danielygo/1781777706/
Marc A. Garrett www.flickr.com/since1968/9923894/
Cortia CenaCarioca www.flickr.com/cenacarioca/355888627/
Cheon Fong Liew www.flickr.com/liewcf/894035077/
Luca Mascaro www.flickr.com/lucamascaro/523822075/
Additional thanks to:

Wonderbrains.com

Apple.com

OLPC

Perceptive Pixel
How Bodies Matter: Five Themes for Interaction Design

Scott R. Klemmer, Bjrn Hartmann, Stanford University HCI Group,

Computer Science Department
Leila Takayama, Stanford University CHIMe Lab, Communication
Department
DIS 2006, June 2628, 2006, University Park, Pennsylvania, USA.
Copyright 2006 ACM 1-59593-341-7/06/0006.
http://hci.stanford.edu/publications/2006/HowBodiesMatter-
DIS2006.pdf
Permission to make digital or hard copies of all or part of this
work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or
commercial advantage.
Dan Saffer, reviewing the paper

http://www.odannyboy.com/blog/new_archives/2008/05/
review_five_the.html

All the rest

http://wikipedia.org

University of Applied Sciences Constance, Faculty for

Communication Design, Project Touch Research

http://www.htwg-konstanz.de
http://www.kd.fh-konstanz.de/dina8/daten_e.php?wodenn=will
http://www.felgner.ch/2008/04/touch_research.html