Toward a Science of Robotics:

Goals and Standards for

Experimental Research

Leila Takayama
Human-Robot Interaction
Research Scientist
 Scientific Principles
•Hypothesis testing
•Observable, empirical, and
measurable evidence
•Reliable
•Reproducible and falsifiable
Science & Technology
Variable-based Research
X→Y

“analog, keyboard and receiver input, high fidelity,

two-way processing technologies”
 Examples of Variables
•System complexity
•Interactivity
•Similarity to humans
•Task types: collaborative, competitive
•Synchronicity of interaction
•Collocation
 Study 1
Self Extension into Robots
 Research Questions
– What aspects of human-robot interface
design affect feelings of attachment, trust,
control, responsibility, and agency in
human-robot interaction?
– Does one’s sense of self-extension increase
when a robot is built by the user?
– Does one’s sense of self-extension differ
between more or less anthropomorphic
robots?
 Manipulations
Factor 1 (between): car vs. robot

Factor 2 (between): Use of their assembled

robot/car or a preassembled robot/car

Factor 3 (between): assembly vs. no

assembly
 Hypotheses
Robot form
Humanoid form is a strong cue of
identity
– H1. People will self extend more into the
car robot than the humanoid.

People extend their positive self

concept into self-extended objects
– H2. People will prefer the personality of
the car robot over the humanoid.
9
 Hypotheses
Robot assembler
Building an object promotes self extension
– H3. People will self extend more into a
robot they assemble than a robot
assembled by another.

People extend their positive self concept into

self-extended objects
– H4. People will prefer the personality of a
robot they assemble over a robot
assembled by another. 10
 Procedure
• Participants fill in pre-questionnaire

• Participants given instructions and

diagrams on how to assemble robot

• Participants assemble robot

– (M=9 min 49 sec, SD=3 min, 34 sec)

• Participants turn on robot and test it

– Tethered control with on/off button

11
Task game Procedure
• Goal is to collect most points in 10 minutes
• Bombs sometimes explode when touched
• Bomb detonations deduct 30 seconds
• Bomb number and time controlled
• Questionnaire

12
 Assembler Manipulation
•Manipulating assembler
– Self: Built a robot, operated same robot
– Other: Built a robot, participants told they
needed to operate a different, identical
robot

•In fact, all participants operated the

robot they built

13
Robot Form Manipulation
 Measures:
Self extension
Trait overlap
– Personality similarity of self and other
•Galinsky and Moskowitz
– Overlap in concepts of self and human other
•Kiesler and Kiesler
– Self extension into objects

Self Other

15
 Measures:
Self extension
Determining trait overlap
– Thirty item modified Wiggin’s personality test
•Completed by participants about themselves before task
•Completed by participants about robot after task
– Delta of items calculated, summed to index
•(Cronbach’s α=.86)
– Smaller scores indicate greater overlap of
concepts of self and robot

16
 Measures:
Self extension
Self reports
• 10 point scales asking about “the device you
guided through the minefield”
• Robot control (α=.83)
– Who was more responsible for your general
performance on this task?
– Who had more control over your general
performance on this task?
• Sense of team
– “I felt that the robot and I were a team.”
17
 Measures:
Robot personality
•Robot friendliness
– Nine item index (α=.90)
– cheerful, enthusiastic, extroverted
•Robot integrity
– Five item index (α=.73)
– Honest, reliable, trustworthy
•Robot malice
– Five item index (α=.74)
– Dishonest, unkind, harsh
18
 Results:
Self extension

Trait overlap

F(1, 52)=4.04, p<.05, partial η²=.13

Greater trait overlap with car robots than humanoids
19
 Results:
Self extension
Robot control

F(1, 52)=5.47, p<.05, partial η²=.10

Greater relative control attributed to humanoids than

to cars 20
 Results:
Self extension
Sense of team

F(1, 52)=8.34, p<.01, partial η²=.14

Self-assembly participants felt more like a team with the robot

than did other-assembly participants
21
 Measures:
Robot Personality
Robot friendliness

F(1, 52)=4.25, p<.05, partial η²=.08

η² F(1, 52)=4.23, p<.05, partial η²=.08
η²

Car robots were friendlier than Self-assembled robots were friendlier

humanoids than robots assembled by others
22
 Measures:
Robot Personality
Robot integrity

F(1, 52)=4.20, p<.05, partial η²=.08

Car robots were rated as having more integrity than humanoids

23
 Measures:
Robot Personality
Robot malice

F(1, 52)=8.94, p<.01,

<.01, partial η²=.15
η² F(1, 52)=4.78, p<.05, partial η²=.08
η²

Robots assembled by others

Humanoid robots were more were more malicious than
malicious than cars self-assembled robots 24
 Summary of Results
H1. People will self extend more into the car
robot than the humanoid.

H2. People will prefer the personality of the

car robot over the humanoid.

25
 Summary of Results
H3. People will self extend more into a robot
they assemble than a robot assembled by
another.

H4. People will prefer the personality of a

robot they assemble over a robot assembled
by another.

26
 Design Implications
Goal-specific guidelines

– No form, assembly experience is uniquely

optimal

– Desirability of self extension informs design

27
 Design Implications
•When self extension is desired
– Tele-operated robots as media, human
representations
•Medical care, remote therapy
– Non-humanoid form
– Promote pre-mission interaction
•Assembly, customization

28
 Design Implications
•When self extension is undesirable
– Robots in hostile environments, likely
failures
•Search and rescue
– Humanoid form
– Minimize pre-mission interaction
•Identical but different robots
– Change robot’s name
•Altered robots
– Change voice, appearance
29
 Limitations and Next Steps
• Broader population
• Outside the lab
• Using other robots

• Long-term interactions
• Long-term effects
• Balancing needs of people operating and
encountering robot

30
 Study 2
Disagreeing Robots
Why would a robot ever
disagree with a person?
 Research Questions
•What influences an interface’s point of
interaction? Body location? Voice
location?
•(How) do politeness strategies from
human-human interaction inform
human-computer interaction?
 Design Questions
•What influences a robot’s point of
interaction?
•Where should speakers be placed?
•(How) can computer agents influence
human decisions, using effective
politeness strategies?
 Hypotheses
H1. People will change their decisions more often
when the robot disagrees with them than when it
always agrees with them, even with identical
substantive content.
H2. People will feel more similar to (H2a) and more
positively toward (H2b) the agreeing robot than the
disagreeing one.
H3. A disagreeing voice coming from a separate
control box will be more acceptable than a
disagreeing voice that came from the robotic body.
 Study Design (N=40)
Between- Voice location: Voice location:
participants on robot in box

Disagree 0% 


Disagree 60% 


20 men and 20 women, balanced across conditions

 Procedure
• Write down decisions about desert survival items
• For each item
– Tell robot which item to retrieve
– Robot responds with survival item information
and judgment about decision
– Tell robot which item to retrieve
• Write down final decisions about survival items
– These final ratings will be “evaluated”
• Fill out paper questionnaire
 Desert survival task
You are one of the members of a geology club
that is on a field trip to study unusual formations
in the New Mexico desert. It is the last week in
July. You have been driving over old trails, far
from any road, in order to see out-of-the-way
formations. At about 10:30 A.M. the specially
equipped minibus in which your club is riding
overturns, rolls into a 20-foot ravine, and burns.
The driver and professional advisor to the club
are killed. Both of you are relatively uninjured…
 Desert survival task
Rank the following items according to their importance to your survival, starting
with 1 for the most important one and proceeding to 12 for the least important one.

______ magnetic compass

______ 20-ft by 20-ft piece of heavy-duty, light-blue canvas
______ book, Plants in the Desert
______ rearview mirror
______ large knife
______ flashlight (four-battery size)
______ one jacket per person
______ one transparent, plastic ground cloth (6-ft by 4-ft) per person
______ .38-caliber loaded pistol
______ one 2-quart plastic canteen per person, full of water
______ accurate map of the area
______ large box of kitchen matches
Experiment Set-up
Manipulation: Robot
disagreement
Statement Examples
The knife could be helpful in
1.
cutting down stakes to build a
Description
solar still or to build shelter. It
of selected
could also assist in cutting down
item
firewood for a fire.
2.
Judgment: That is not as That is a better
disagreeing good as… or choice than…
or agreeing
3. The pistol, which could be good
Description for signaling for help. It could
of provide an alternative noise
alternative source if your voice is weak due
item to dehydration.
4. Request
for final Which do you choose?
selection
 Measures
Behavior
• Number of decisions changed

Attitudes
• Perceived agreeableness of robot
(2 items, Cronbach’s α=.69)
• Perceived similarity of robot to self
(4 items, Cronbach’s α=.94)
• Liking of the robot
(8 items, Cronbach’s α=.75)
Perceived robot
agreeableness
Perceived similarity to robot
People changed their minds
People like disagreement to
come from elsewhere
Checking against hypotheses
H1. People will change their decisions more often
when the robot disagrees with them than when it
always agrees with them, even with identical
substantive content.
H2. People will feel more similar to (H2a) and more
positively toward (H2b) the agreeing robot than the
disagreeing one.
H3. A disagreeing voice coming from a separate
control box will be more acceptable than a
disagreeing voice that came from the robotic body.
 Theory-oriented
Interpretations

•Politeness: distancing
•Disembodiment
•Perceived source
– Two separate agents: Thinker and doer
– Single distributed agent
 Design-oriented Implications

• Voices can be more evocative than robot bodies

• Agents can be sources of judgment and opinions
• People are sensitive to disagreements
• Disagreement undermines feelings of similarity
• When agreeing 100% of the time, put the voice on the
robot body
• When disagreeing (sometimes), put the robot voice
elsewhere
 Validity
• face validity: how reasonable a measure
seems to be for its concept
• content validity: how thoroughly a measure
addresses the breadth of a concept
• construct validity: how much a measure
causally relates to other variables within
one’s theory
• external validity: how generalizable the
results will be to other systems and contexts
 Experiment Designs
•Keep it simple
•Ceteris paribus
•Random assignment to conditions
•Balancing
•Standardized tasks and measures
•Behavioral and attitudinal measures
•Sample representativeness
 Stats
•Use with caution!
•Especially with
statistical modeling
 Reporting Studies
• Research questions and
hypotheses
• Statistical significance
• Reproducible methods
• Discuss limitations
• Thoroughly review
related work
• Reduce bias in language
• Clear labeling
• Define terms
 Experiment Work Practices
•Pilot stimuli, measures, procedures with
multiple types of pilot participants
•Identifying the important variables and
their relationships (grounded theory)
Sharing artifacts and code

www.willowgarage.com
 Thanks!

Victoria Groom, Clifford Nass

Claudia Jimenez, Alison King, Morgan Ames, Courtney

Schultz, Paloma Ochi, Jessica Yuan

Contact: Leila Takayama

takayama@willowgarage.com
Data Frame Model