Interacting with Computers 19 (2007) 429–437

www.elsevier.com/locate/intcom

Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021

To do or not to do: Differences in user experience and retrospective
judgments depending on the presence or absence
of instrumental goals
a,* b
Marc Hassenzahl , Daniel Ullrich
a
Economic Psychology and Human–Computer Interaction, University of Koblenz-Landau,
Fortstraße 7, 76829 Landau, Germany
b
Darmstadt University of Technology, Social Psychology and Decision Making, Alexanderstraße 10,
64283 Darmstadt, Germany

Received 2 February 2007; received in revised form 30 April 2007; accepted 14 May 2007
Available online 21 May 2007

Abstract

Recently, Human–Computer Interaction (HCI) started to focus on experiential aspects of product use, such as affect or hedonic qual-
ities. One interesting question concerns the way a particular experience is summarized into a retrospective value judgment about the
product. In the present study, we specifically explored the relationship between affect, mental effort and spontaneity experienced while
interacting with a storytelling system and retrospective judgments of appeal. In addition, we studied differential effects of the presence or
absence of instrumental goals. In general, active instrumental goals did not only impact experience per se by, for example, inducing men-
tal effort, but also the way subsequent retrospective judgments were formed. We discuss the implications of our findings for the practice
of product evaluation in HCI specifically, and more general aspects, such as the role of affect in product evaluations and the importance
of usage mode compatibility (i.e., a compatibility of the way one ought to and actually does approach a product).
 2007 Elsevier B.V. All rights reserved.

Keywords: User experience; Affect; Evaluation; User satisfaction; Task; Instrumental goals; Context-dependency; Goal-mode; Action-mode

1. Introduction state of affairs (e.g., Kahneman, 1999). In other words, it

Imagine using a Web site to look for a Christmas pres-
ent. During your quest, you are likely to experience vari-
ations in the quality of the service. You will encounter
usability problems as well as good suggestions by the
Web site’s recommendation system. And with a bit of
luck, this particular usage episode will end with a perfect
present and the successful completion of the check out
procedure.
The subjective representation of such a usage episode is
an experience. During an experience, the brain continu-
ously constructs an affective commentary on the current
* the experience and the product. Moreover, evaluations will
Corresponding author. Tel.: +49 6341 280261.
E-mail addresses: hassenzahl@uni-landau.de (M. Hassenzahl), ullrich
@psychologie.tu-darmstadt.de (D. Ullrich).
is a stream of valuable and not so valuable moments with
a definite beginning (e.g., the opening of a good bottle of
wine) and ending (e.g., the last sip of the wine).
During an experience individuals can probe their
momentary state. By that, the state becomes conscious.
Moreover, individuals are able to make summary judg-
ments about the overall quality of an experience and ele-
ments of it in retrospect (Kahneman, 1999; Hassenzahl
and Sandweg, 2004). In the field of Human–Computer
Interaction (HCI) any subjective product evaluation pro-
vided by users after a product trial is an instance of retro-
spective summary evaluation. For users/consumers those
evaluations are the basis for further communicating about
be memorized and, by that, will presumably guide future
behavior.

0953-5438/$ - see front matter  2007 Elsevier B.V. All rights reserved.
doi:10.1016/j.intcom.2007.05.001
430 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
Often, researchers and practitioners in the field of HCI
take retrospective summary evaluations as direct indicators
of product quality. Their implicit assumption is that retro-
spective evaluations are simple averages or sums of all the
moments encountered during the experience. Research,
however, reveals a more complex picture. In a well-known
study on colonoscopy (Redelmeier and Kahneman, 1996),
for example, patients were asked to continuously indicate
the intensity of pain experienced during the procedure.
After the procedure they provided a retrospective judgment
about how painful the procedure was in general. Those
judgments were best predicted by a peak-end rule, i.e., an
average of the peak intensity and the end intensity of pain
experienced during the episode. Similar results were found
for less extreme situations, such as retrospective summaries
of the affect experienced while watching a movie (Freder-
ickson and Kahneman, 1993). In the context of HCI, Has-
senzahl and Sandweg, 2004 studied the relation between
experience and retrospective product evaluations. Partici-
pants of a usability test worked through a series of inde-
pendent tasks. After each task they were asked to rate
the mental effort experienced during the task. Effort served
as a proxy for encountered usability problems (Hassenzahl,
2000). Subsequently, they were asked to rate the product’s
usability. As expected, more effort led to less favorable
evaluations of the product. However, the effort experienced
in the last task was a better predictor of the participants’
evaluation than any other indicator (e.g., average of all
effort measures, peak/end, variance, trend). Thus, the ret-
rospective usability evaluation was related to, but not iden-
tical with the experience participants made. It was rather
constructed on the spot, based on available information,
either derived directly from the target (e.g., by taking sali-
ent moments from the experience into account) or from
general knowledge about and attitudes toward the target
(see Kardes et al., 2004).
A further aspect that impacts retrospective product eval-
uations is the situation. In a study, Hassenzahl et al., 2002
found tasks to impact the way participants formed overall
judgments of how appealing a number of Web sites were.
Participants were either given tasks to accomplish with
the Web sites (e.g., to find a particular information) or
the instruction to just ‘‘have fun with the Web sites.’’ For
the task-group, subjective perceptions of usability (e.g.,
simple–complex) were strongly correlated with appeal
(e.g., good–bad) (partial correlation = .87), which was not
apparent for the fun-group (partial correlation = .10).
In the former case, usability as a product attribute was pre-
dictive of the product’s overall appeal, in the latter not.
Depending on the context, the basis of the product evalua-
tion changed.
The present paper examines the relation between mea-
sures taken during an experience and retrospectively. Spe-
cifically, we let participants assess the valence of their
momentary feelings (i.e., affect) and exerted mental effort
during the interaction with a digital storytelling system
based on the art-E-fact platform (Iurgel, 2004a,b; Spier-
ling, 2005). Affect was included, because of its general
importance to judgmental processes. Spontaneous good
or bad feelings often serve as a basis for judgments on var-
ious aspects of tasks or products (Schwarz and Clore, 1983,
see Pham, 2004, for an overview). Experienced mental effort
is the amount of energy a user has to activate to meet
perceived task demands (Arnold, 1999). It is a predictor
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
for encountered usability problems (Hassenzahl, 2000),
thus, related to experienced barriers in goal attainment.
After the experience, participants were asked to assess their
spontaneity while interacting with the product (I considered
my actions carefully – I decided spontaneously for actions),
the product’s overall appeal and to give positive and nega-
tive comments on the product/the experience. In addition,
we varied the situation by either inducing an external
instrumental goal for the interaction with the product or
not. Providing an external goal was expected to change
the way experiential measures relate to retrospective
measures.
2. Method
2.1. Participants
Thirty individuals (13 female, median age = 26.5,
min = 20, max = 51) participated in the study. The major-
ity were students of the Darmstadt University of Technol-
ogy or employees of the Computer Graphics Center
Darmstadt (ZGDV). They received no compensation for
participation.
2.2. Stimulus product: art-E-fact
Art-E-fact is a generic platform for the creation of inter-
active stories developed by the Computer Graphics Center
Darmstadt (ZGDV) (Iurgel, 2004a,b; Spierling, 2005, see
www.art-e-fact.org for further information). It consists of
an authoring tool for the generation of stories (e.g.,
story-line, virtual characters, interaction rules) and a set-
up for viewing and interacting with the story. In the present
configuration, the story was displayed by a data projector
onto the wall (2 by 1.50 m). During the experiment, partic-
ipants were standing in front of the wall, watching and
interacting with the story.
We used a story generated for a museum environment.
While the story’s plot about an art theft unfolded, users
were presented with a number of different paintings, back-
ground information on artists, the analysis of art and art in
general. The story was driven mainly by dialogues between
two synthetic characters: a reporter and a knowledgeable
art professor. All characters spoke with synthesized voices,
with matching gestures and facial expressions.
At particular turns of the story, participants could
interact with the system. The basic interaction mechanism
was pointing. A camera-based recognition system captured
the hand-movement and extrapolated where users pointed
at. This mechanism was used to activate specific functions,
 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
introduced and explained by the characters, such as skip-
ping or repeating parts of the story or getting additional
information about particular paintings. In addition, the
participants could use purpose-built tools, such as a magni-
fying glass, to enlarge portions of pictures or to reveal hid-
den layers of paint.
2.3. Independent variable: instrumental goal
A major objective of the present study was to explore
differences in the user experience and relations to retrospec-
tive judgments depending on the situation. Specifically, the
situation was varied by either letting participants experi-
ence the system freely at their own pace (no-goal condition)
or by asking them to find answers to a number of particular
questions with the help of the system (goal condition).
Finding particular pieces of information was selected as
the instrumental goal (i.e., task) because acquiring infor-
mation about art was supposed to be a major outcome of
the implemented story. Note that the present experiment
requires a product that can either be experienced with or
without active instrumental goals. Most products studied
in HCI are utilitarian systems, build for a particular pur-
pose. Art-E-fact, however, can be used either explorative
and for the sake of the story or as a source of information
about paintings. This twofold nature makes it a particular
interesting stimulus product.
2.4. Experiential measures
2.4.1. Mental effort
Mental effort can be defined as the amount of energy a
user has to activate to meet the perceived task demands
(Arnold, 1999). This energy depends on objective demands
of the task and the individuals’ preparedness and coping
strategies. Thus, mental effort is subjective and always tied
to a task, i.e., meaningful instrumental goals. The present
study used the ‘‘Subjective Mental Effort Questionnaire’’
(SMEQ, Zijlstra, 1993; Eilers et al., 1986), a simple rating
scale ranging from 0 (hardly effortful) to 220 (exceptionally
effortful). Studies showed this measure to be related to sub-
jective ratings of product usability, appeal (Hassenzahl,
2001) and time spent to correct usability problems (Hassen-
zahl, 2000). Unpublished data further showed a substantial
correlation between the number of usability problems
observed by a usability expert and the SMEQ rating (aver-
aged r = .47).
In the present study, we were rather interested in an
experiential than in a retrospective measure of mental
effort. To achieve this, we measured mental effort three
times throughout the usage episode (see Section 2.6). The
internal consistency of the three separate measurements
was satisfactory (no-goal: Cronbach’s a = .78; goal: Cron-
bach’s a = .84). A 2 · 3 analysis of variance with situation
(no-goal, goal) as between-subjects factor, time (1., 2., 3.)
as within-subjects factor and mental effort as dependent
variable showed no significant main effect of time or time
431
by goal interaction. We averaged all three measurements,
following the logic of Kahneman (1999) to obtain an
‘‘objective’’ experiential measure of mental effort.
2.4.2. Affect
Affect can be defined ‘‘as a neurophysiological state that
is consciously accessible as a simple non-reflective feeling
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
[. . . This feeling] is an integral blend of hedonic (plea-
sure–displeasure) and arousal (sleepy–activated) values’’
(Russell, 2003, p. 147). For the present study, we restricted
our analysis to valence – that is the pleasure and displea-
sure – experienced during interaction with the system as
one of the key dimensions of virtually all models of affect
(Russell, 2003). Valence was measured with the ‘‘Self
Assessment Manikin’’ (Bradley and Lang, 1994). Similar
to mental effort, three separate measurements were taken.
The internal consistency was satisfactory (no-goal: Cron-
bach’s a = .90; goal: Cronbach’s a = .81). A 2 · 3 analysis
of variance with situation (no-goal, goal) as between-sub-
jects factor, time (1., 2., 3.) as within-subjects factor and
affect (valence) as dependent variable showed a significant
main effect of time, F (2, 56) = 3.68, p < 0.05, best described
as a U-form shape, i.e., a dip in affect for the mid measure-
ment. However, no significant time by goal interaction
emerged. Thus, we averaged all three measurements to
obtain an experiential measure of affect.
2.5. Retrospective measures
2.5.1. Evaluation
Seven seven-point items with bipolar verbal anchors
were used to capture the retrospective evaluation of the sys-
tem. The scale is a part of the AttrakDiff2-questionnaire
(Hassenzahl et al., 2003) and was already used in a number
of different studies (e.g., Hassenzahl, 2001) to measure the
appeal of interactive products. Example items are good–
bad, inviting–rejecting or likable–disagreeable. The internal
consistency of the scale was satisfactory (no-goal: Cron-
bach’s a = .95; goal: Cronbach’s a = .94). Evaluation
(i.e., appeal) was calculated as the average of all seven
items.
2.5.2. Acquired knowledge
To measure the amount of acquired knowledge, partic-
ipants had to answer four questions after finishing the
story. All required information was provided in the main
story-line, thus, individual choices regarding different story
threads or following up additional information cannot be
made responsible for differences in the acquired knowledge.
The four questions were: (1) ‘‘Which was the university
James [a reporter] visited to gather information about the
stolen painting?’’ (2) ‘‘Name the two artists whose paint-
ings were on display,’’ (3) ‘‘What was Guardi’s [a Venetian
artist] profession?,’’ and (4) ‘‘What was the name of the
television program the professor wanted to watch?’’. Ques-
tions were designed to ensure that participants could not
rely on previous knowledge.
432 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
2.5.3. Spontaneity
Hassenzahl (2003; Hassenzahl, Kekez, & Burmester,
2002) argued that users can be either in goal- or action-
mode while interacting with an interactive product. In the
goal-mode the current instrumental goal has a certain
importance and determines more or less all activities. The
system is ‘‘a means to an end.’’ Besides other aspects, indi-
viduals tend to plan ahead, i.e., to consider their actions
carefully with respect to active goals. In the action-mode
the current activity determines goals ‘‘on the fly;’’ goals
are ‘‘volatile.’’ Using the system is an ‘‘end in itself.’’ Indi-
viduals rather spontaneously decide their actions, because
goals will be made up in the course of the interaction
instead of guiding the action as in goal-mode.
We tap into participants’ differences in mode by letting
them rate their felt ‘‘spontaneity’’ during the experience
in retrospect on a seven-point scale with the bipolar verbal
anchors I considered my actions carefully – I decided spon-
taneously for actions taken from Hassenzahl and colleagues
(2002).
2.5.4. Positive/negative comments
In a debriefing interview, participants were asked to
comment freely on positive and negative aspects of the sys-
tem. Specifically, we asked them to think of their experi-
ence with the product and to report three particularly
positive and three particularly negative aspects of the prod-
uct or their experience.
2.6. Procedure
Participants were led separately into the laboratory.
During the experiment, they stood on a mark approxi-
mately 2.50 m in front of the image displayed by a data
projector in the size of 2 by 1.50 m. The laboratory was
shaded, the display remaining the only light source.
The experimenter started with a short introduction to
art-E-fact’s main features, such as using one’s finger as a
pointing device and the resulting interaction. After a very
brief introduction to the story, participants in the no-goal
condition were instructed to just have fun with the story
and to do with the product whatever they like. Participants
in the goal condition were told to explore the story and to
find answers to four questions on story details and artists’
biography (see Section 2.5.2). After participants indicated
that they had understood all instructions, the story was
started.
During the experiment, the story was interrupted three
times at scene-transitions to let participants assess their
momentary affective state (see Section 2.4.2) and the men-
tal effort experienced right at that moment (see Section
2.4.1). Scene-transitions provided ‘‘natural’’ moments for
interruptions, thereby keeping the impact of the interrup-
tion itself on the current affective state at a minimum
(e.g., Adamczyk and Bailey, 2004). The questionnaires
were filled in at the position, where participants interacted
with the system. The first measurement took about 45 s,
due to a short introduction given by the experimenter.
The remaining two measurements took about 15 s each.
On average, participants interacted 34 min with the story,
with a minimum of 25 min and a maximum of 45 min.
After having gone through the story, participants took a
5 min break. After this, they were first asked to assess their
spontaneity while using the system (see Section 2.5.3). Sub-
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
sequently, participants answered the four open knowledge
acquisition questions (see Section 2.5.2). For the goal
group these questions were already known, and partici-
pants simply wrote down the information they remembered
from interacting with the story. For the no-goal group the
questions were new. Finally, each participant evaluated the
product (see Section 2.5.1). The experiment ended with a
debriefing interview. Participants were asked to point out
up to three particularly positive and three particularly neg-
ative aspects of the product or their experience (see Section
2.5.4).
2.7. Predictions
Although the present study is mainly explorative some
predictions concerning the different experiential and retro-
spective variables can be made. First, the average number
of correct answers given to the four knowledge acquisition
questions should be higher in the goal compared to the no-
goal condition. This is due to the explicit setting of the
knowledge acquisition goal and the fact that the questions
were known in advance (i.e., during interaction).
Second, mental effort should be higher in the goal com-
pared to the no-goal condition. Mental effort is closely tied
to the attainment of instrumental goals, i.e., to tasks
(Arnold, 1999). If no goal is made salient, goal-directed
behavior and general problems with accomplishing goals
(expressed as mental effort) become less likely.
Third, spontaneity should be higher in the no-goal con-
dition compared to the goal condition. Hassenzahl and col-
leagues (2002) argued that a ‘‘have fun’’-instruction (as in
the no-goal condition) results in an action mode, whereas
the explicit setting of goals (as in the goal condition) results
in a goal mode. One particular difference between both
modes is the amount of spontaneity users experience dur-
ing interaction.
In addition, particular relations between the different
variables were expected. First, mental effort should be
related to knowledge acquisition because effort is tied to
goal-directed behavior and the number of correct knowl-
edge acquisition questions is the actual measure for goal
attainment. Second, the basis of product evaluation (i.e.,
appeal) should vary with the situation: in the goal condi-
tion, absence of mental effort and good knowledge acqui-
sition are supposed to be important predictors of the
overall product evaluation. In the no-goal condition, how-
ever, both should lose their importance. Third, affect
should be related to all variables, emphasizing the impor-
tance of affective responses as input for any judgmental
process.
 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437 433

3. Results and discussion
3.1. Experiential and retrospective measures
Table 1 (column 2) shows the overall mean and standard
deviation of each measure. On the whole, interacting with
art-E-fact was experienced as to some degree effortful,
To further explore the relationships among the experien-
tial (mental effort, affect, spontaneity) and retrospective
measures (evaluation, knowledge acquisition), we per-
formed two separate principal components analyses
(PCA, extraction criterion: Eigenvalue >1, Varimax rota-
tion) on each condition (no-goal, goal). Gorsuch, 1997 rec-
ommends at least a ratio of three independent observations

Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021

but accompanied by positive affect. The system was evalu-
ated as rather appealing. Knowledge acquisition was fair
and the participants used the system in a noticeably spon-
taneous way.
Table 1 (column 3 and 4) further shows the mean and
standard deviation of each measure for the no-goal and
goal condition. Significant differences between no-goal
and goal condition emerged for mental effort and knowl-
edge acquisition (see Table 1, column 5), with a higher level
of mental effort and better knowledge acquisition in the
goal condition. Both was expected and can be taken as evi-
dence for the successful manipulation of the situation. The
finding that the induction of the knowledge acquisition
goal led to an increase in mental effort underlines the close
relationship between mental effort and goal-directed behav-
ior. It further gives an insight into the way usability prob-
lems were experienced. As long as participants in both
conditions used the same features of the product for
approximately the same amount of time, both groups were
equally likely to encounter usability problems. However,
those ‘‘objective’’ usability problems led to an increase in
effort only for those participants with an externally
imposed goal (to acquire information).
for each included variable, which was given in the present
study. To rule out the possibility that differences in the
emerging patterns of relations among variables are due to
large differences in the variability of those variables, tests
of the homogeneity of variance were performed for each
variable (see Table 1, column 6). No significant differences
in variances emerged.
Table 2 (column 2 and 3) shows the result of the PCA
for the no-goal condition. Two components with an Eigen-
value greater than 1 were extracted. Both components
together explained 73% of the total variance. Positive affect
and spontaneity, together with the retrospective evaluation
of the product, formed one component; low mental effort,
positive affect and knowledge acquisition the other compo-
nent. This pattern disappeared in the goal condition (Table
2, column 4): only one component was extracted (60%
explained variance) with loadings of all variables.
Several interesting differences between the patterns of
the two conditions (no-goal, goal) emerged. Most strik-
ingly, in the goal condition experienced mental effort was
highly related to the retrospective evaluation of the system
(direct r = .80, p < .001); a relation, which was reduced in
the no-goal condition (direct r = .47, n.s.). This supports

Table 1
Mean (standard deviation), t-tests and homogeneity of variances for all measures
Measure (scale) Overall No-goal (N = 15) Goal (N = 15) t(28) Levene F
*
Mental effort (SMEQ) (0–220) 78.67 (41.96) 61.12 (34.47) 96.23 (42.41) 2.49 .23
Affect–valence (SAM) (1–9) 6.37 (1.40) 6.52 (1.33) 6.23 (1.50) .56 .04
Evaluation (APPEAL) (1–7) 5.17 (1.10) 5.18 (1.12) 5.15 (1.11) .07 .01
Knowledge acquisition (0–4) 2.03 (1.50) 1.33 (1.35) 2.73 (1.34) 2.86** .02
Spontaneity (I considered my actions carefully – I 5.60 (1.61) 5.67 (1.40) 5.53 (1.85) 0.22 .78
decided spontaneously for actions) (1–7)
*
p < .05.
**
p < .01.

Table 2
Factor loadings for the no-goal and goal condition
No-goal (N = 15) Goal (N = 15)
Experiential 1 2 1
Mental effort .752 .890
Spontaneity (I considered my actions carefully – I decided .866 .553
spontaneously for actions)
Affect–valence .572 .341 .943
Retrospective
Knowledge acquisition .918 .472
Evaluation (APPEAL) .952 .891
Eigenvalue 2.11 1.55 3.03
Explained variance 42 31 60
Notes: Principal components analysis; Extraction criterion: Eigenvalue >1, Varimax rotation, all loadings <.300 were suppressed.
434 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
the notion of a context-dependent evaluation process, i.e.,
an ‘‘on the spot’’ construction of the evaluation. In other
words, participants ground their evaluation on variables
predictive for a given context, such as mental effort in the
context of the goal condition.
A further interesting difference concerned the relation of
spontaneity to evaluation. Whereas in the no-goal condi-
tion experienced spontaneity was highly predictive for a
favorable evaluation (direct r = .76, p < 0.001), this rela-
tion disappeared in the goal condition (r = .33, n.s.).
Remarkably, the relation changed its direction (sign):
whereas in the no-goal condition more spontaneity was
related to more favorable evaluations, in the goal condition
less spontaneity (i.e., more planning) led to an in tendency
more favorable evaluation. This underlines the importance
of the compatibility between the mode (goal, action) of the
individual (i.e., actual mode) and the requirements of the
situation at hand (i.e., appropriate mode).
Not surprisingly, affect is context-dependent just as eval-
uation is. In the no-goal condition positive affect was asso-
ciated with high spontaneity, whereas in the goal condition
affect becomes rather associated with mental effort. This
illustrates the notion that experiences can considerably
vary and be still summarized into a simple good/bad-judg-
ment (Kahneman, 1999). Affect is the psychological cur-
rency that makes the comparison of qualitatively different
experiences possible (Russell, 2003). In line with this view,
affect is in general the single best predictor for the retro-
spective product evaluation. A stepwise regression with
mental effort, affect, spontaneity, and knowledge acquisi-
tion as predictors and evaluation as criterion, resulted in
a model with affect as a single predictor (b = 0.69,
R2 = 0.46, F = 25.51, p < 0.001). Thus, in the present
study, affective states during the experience were, if not
caused by, at least attributed to the product. This resulted
in a clear relationship between affect and evaluation of the
product.
The principal components analysis of the goal condition
showed knowledge acquisition to be – albeit weakly – asso-
ciated with low mental effort, positive affect, planning and
a positive evaluation of the product. In the no-goal condi-
tion knowledge acquisition was dissociated from evalua-
tion, simply because knowledge acquisition had not been
explicitly mentioned and thus was not used as a basis for
the evaluation of the product. Interestingly, high knowl-
Table 3
Comment categories, their definition and an example
Category Definition: comments refer to. . .
Voice ... the synthesized voice and its intelligibility
Interaction ... the opportunity of interacting differently with the product
Tools ... the used tools (e.g., magnifying glass)
Pointing ... the products gesture-based pointing-feature
Characters ... the displayed characters
Story ... the story-plot
Novelty ... the product’s novelty and innovation
Fun ... the experienced fun
edge acquisition in the no-goal condition remained related
to low mental effort (they form their own component).
3.2. Positive/negative comments
In the debriefing interview, participants made 93 posi-
tive and negative comments on the product or their experi-
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
ence. Comments were further classified by the authors into
eight broad categories (see Table 3), addressing particular
features of the product (e.g., pointing gesture), content
(e.g., the featured story) as well as high level attributes such
as the fun derived from using the product. Eighty-four per-
cent of comments (78 of 93) could be categorized this way.
The remaining 16% were not classifiable and omitted from
further analyses.
Table 4 shows the absolute and relative frequencies for
comments in the different categories overall, separated by
positive and negative comments as well as for each condi-
tion. The most frequently mentioned category was ‘‘voice’’
(26%, 20 of 78) referring to, for example, voice synchronic-
ity and speed. The categories ‘‘novelty’’ and ‘‘fun’’ received
the fewest mentions (4%, 3 of 78).
A simple way to examine differences between the goal
and the no-goal condition is to analyze the relationship
(i.e., correlation) between category-rankings. The more fre-
quent the comments in a particular category, the lower its
according rank. A low correlation between rankings of
both groups hints at differences between groups, whereas
a high correlation indicates correspondence.
Goal and no-goal rankings (computed on the total num-
ber of comments in the category) did not correlate signifi-
cantly, q = .39, n.s., n = 8. Participants in the goal
condition commented more frequently on ‘‘interaction’’
and ‘‘story’’ issues, whereas participants in the no-goal
condition commented more frequently on ‘‘tools’’. We fur-
ther analyzed differences between conditions for positive
and negative comments separately. Participants in both
conditions agreed on the negative aspects of the product,
q = .88, p < .01, n = 8, but disagreed on the positive
aspects, q = .51, n.s., n = 8. As expected, comments of par-
ticipants in the goal condition revolved around goal attain-
ment, i.e., interaction with the products and problems with
understanding the synthesized voices, whereas participants
in the no-goal condition had no preferred issue – their com-
ments were spread more evenly over all categories.
Example [#participant-number]
‘‘the voice is not intelligible and ridiculous’’ [#12]
‘‘options: you may learn more but you haven’t have to’’ [#6]
‘‘tools had malfunctions’’ [#3]
‘‘pointing is inaccurate and exhausting’’ [#12]
‘‘characters are static and artificial’’ [#30]
‘‘the story is vivid’’ [#12]
‘‘a new type of knowledge transfer’’ [#4]
‘‘it’s playful to use the program’’ [#27]
 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
Table 4
Absolute frequencies (%) of comment categories
Category Overall
Total Positive Negative
Voice 20 (26) 1 19
Interaction 15 (19) 13 2 10
Tools 12 (15) 3 9
Pointing 11 (14) 4 7
Characters 8 (10) 2 6
Story 6 (8) 5 1
Novelty 3 (4) 3 1
Fun 3 (4) 3 1
Total 78 34 44
4. Summary and conclusion
To have an active instrumental goal or not does not only
impact the experience of an interactive product per se, but
also subsequent retrospective judgments. In the present
study, active instrumental goals made barriers to their
attainment salient indicated by an increase in mental effort.
In addition, mental effort was negatively related to affect,
and both acted as input to retrospective product evalua-
tions (i.e., appeal). Goal attainment itself, approximated
by the number of correct answers given to a series of ques-
tions (i.e., knowledge acquisition), was as well related to
product evaluation, albeit not as strongly. Interestingly,
the more spontaneous participants felt during interaction,
the more effort and more negative affect they experienced.
In addition, spontaneity was related to a reduced appeal
of the product.
If no instrumental goal was active, experience and the
relation of experiential variables to retrospective product
evaluations changed. Now, spontaneity was experienced
as positive and was highly related to a positive product
evaluation. Mental effort and knowledge acquisition still
formed a group of goal-related aspects; however, both were
dissociated from product evaluation. As long as mental
effort is related to pragmatic quality perceptions (i.e.,
instrumental quality, usability and utility of a product),
this conceptually replicates the findings of Hassenzahl
et al., 2002 with pragmatic quality being predictive of prod-
uct appeal in a situation with active instrumental goals, but
not in a situation where participants were told to have fun.
The apparent profound impact of active instrumental
goals on the way people experience and evaluate an inter-
active product is especially relevant for HCI, as long as
‘‘the task’’ is one of the pivotal elements of product evalu-
ation (Rubin, 1994). Tasks induce instrumental goals and
focus participants on their attainment. Accordingly, the
product is evaluated in terms of its capability to support
goal attainment (i.e., its usability). However, the present
study shows that this is not necessarily the only way prod-
ucts can be experienced and judged. Without external
instrumental goals given, participants change their set of
criteria as demonstrated by the different roles mental effort
and spontaneity played in the goal compared to the non-
435
Goal No-goal
Total Positive Negative Total Positive Negative
11 11 9 1 8
9 1 5 4 1
3 1 2 9 2 7
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
5 1 4 6 3 3
3 1 2 5 1 4
5 4 1 1 1
1 2 2
1 2 2
39 18 21 39 16 23
goal condition. This highlights how the setting of an eval-
uation impacts its results, an effect broadly called demand
characteristics (Orne, 1969). Cordes, 2002 provided a strik-
ing example of demand characteristics in the context of
usability testing, an effect he dubbed the ‘‘I know it can
be done or you wouldn’t have asked me to do it’’-bias.
He showed participants in a standard usability test setting
to be six times more persistent in accomplishing tasks, if
they were told that they should not assume ‘‘that the prod-
uct can perform each task that we are going to ask you to
do’’ (p. 413). Given this instruction, they were not only
more persistent; they also rated the same tasks as being
14 times less difficult. The present study shows that the
mere evaluation technique itself, i.e. a strong focus on
tasks, alters the set of criteria used and may thus impact
the result of an according evaluation. Note that in the pres-
ent case, appeal did not differ between conditions. How-
ever, this may be simply the consequence of the balanced
nature of the product studied. In other words, art-E-fact
may be relatively appropriate for both types of use, a goal
and an action-oriented. One can easily imagine other inter-
active products, which have to cater for both types of uses.
In this case standard usability testing may be severely
limited.
On a more general level, the study demonstrated not
only the context-dependency of overall evaluative judg-
ments (e.g., Hassenzahl, 2003), but also the pervasive role
of affect for retrospective judgments and the importance
of mode compatibility. In both conditions (goal, no-goal),
affect was related to product evaluations. However, in the
no-goal condition, affect was the only variable with load-
ings on both components (spontaneity/evaluation versus
mental effort/knowledge acquisition). This underlines the
central role of affect in user experience. Interaction with
a product, no matter whether goal or not goal-oriented,
is inevitably accompanied by affect. In both conditions,
for example, mental effort was accompanied by negative
affect, i.e., experienced as negative. In the present case,
the measurement of affect was experiential, i.e., measures
were momentary snapshots of affective states taken from
the interaction. Experiential affect is related to but not
identical with product evaluation. Only experiential affect
attributed to the product will be relevant for a subsequent
436 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
evaluative judgment. This becomes very apparent in the
no-goal condition where only the proportion of affect
related to spontaneity was relevant for evaluation but
not affect related to mental effort. In other words, individ-
uals base their evaluations on affective experience but not
without a further stage of ‘‘editing,’’ where affect has to
be related to the product (i.e., attributed to) to be used.
The experiential character of affect in the present study
distinguishes it from studies, such as Zhang and Li’s
(2004). They showed affective quality to be a predictor
of usability, utility and ultimately of behavioral inten-
tions. Affective quality (Russell, 2003), however, is a con-
struct already closely tied to an object, namely its
perceived ability to change affect. The present study
extends these findings by showing how people selectively
base their evaluative judgments on affective experience
during product use.
Spontaneity during interaction was valued very differ-
ently. Whereas in the no-goal condition, spontaneity was
valued in terms of a positive relation to product evaluation
the opposite was found in the goal-condition. Hassenzahl,
2003 argued for two modes of using an interactive product,
a goal- and an action-mode. In this concept, aspects of user
experience appropriate for one mode may be inappropriate
for the other. In other words, the same experiential quality,
such as the spontaneous interaction with a product, may be
appreciated in one mode but spurned in the other. The
present study supports this notion. The same level of spon-
taneity was either experienced as appropriate, i.e., accom-
panied by positive affect and a favorable product
evaluation or as inappropriate, i.e., accompanied by nega-
tive affect and an unfavorable product evaluation. This
point at the importance of what we call mode compatibility,
i.e., a compatibility of the way one ought to and actually
does approach a product.
The present study is certainly limited in scope. Obvi-
ously, only a single product was studied, with a limited
number of people, which may render the broader gener-
alization of results problematic. Note, however, the
exploratory nature of the study and the unique opportu-
nity to study an experimental interactive product explic-
itly designed for both, a goal-oriented (i.e., knowledge
acquisition) and an action-oriented mode (i.e., story,
playful exploration). Future studies may further expli-
cate and specify the relationship between experience
and retrospective product evaluations as well as its
moderation by different usage situations. This would
surely be important contributions to the growing field
of user experience research (Hassenzahl and Tractinsky,
2006).
Acknowledgements
We are grateful to Dr. Stefan Göbel, Anja Hoffmann,
Ido Iurgel, and the whole team of the Computer Graphics
Center Darmstadt (ZGDV) for supporting the present
study. The art-E-fact project ‘‘Generic Platform for the
Creation of Interactive Art Experience in Mixed Reality’’
was funded by the European Union in the context of the
Fifth Framework Program (IST-2001-37924). For further
information visit: www.art-e-fact.org.
References
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
Adamczyk, P.D., Bailey, B.P., 2004. If not now, when? The effects of
interruption at different moments within task execution. In: Proceed-
ings of the CHI 04 Conference on Human Factors in Computing
Systems. ACM, New York, pp. 271–278.
Arnold, A.G., 1999. Mental effort and evaluation of user interfaces: a
questionnaire approach. In: Bullinger, H.-J., Ziegler, J. (Eds.),
Proceedings of the HCII ’99 international conference on human–
computer interaction, vol. 1. Lawrence Erlbaum, Mahwah, NJ, pp.
1003–1007.
Bradley, M.M., Lang, P.J., 1994. Measuring emotion: the self-assessment
manikin and the semantic differential. Journal of Behavior Therapy
and Experimental Psychiatry 25, 49–59.
Cordes, E.R., 2002. Task-selection bias: a case for user-defined tasks.
International Journal of Human–Computer Interaction 13, 411–420.
Eilers, K., Nachreiner, F., Hänecke, K., 1986. Entwicklung und Überprü-
fung einer Skala zur Erfassung subjektiv erlebter Anstrengung.
Zeitschrift für Arbeitswissenschaft 40, 215–224.
Frederickson, B.L., Kahneman, D., 1993. Duration neglect in retrospec-
tive evaluations of affective episodes. Journal of Personality and Social
Psychology 65, 45–55.
Gorsuch, R.L., 1997. Exploratory factor analysis: its role in item analysis.
Journal of Personality Assessment 68, 532–560.
Hassenzahl, M., 2000. Prioritising usability problems: data-driven and
judgement-driven severity estimates. Behaviour & Information Tech-
nology 19, 29–42.
Hassenzahl, M., 2001. The effect of perceived hedonic quality on product
appealingness. International Journal of Human–Computer Interaction
13, 479–497.
Hassenzahl, M., 2003. The thing and I: understanding the relationship
between user and product. In: Blythe, M., Overbeeke, C., Monk, A.F.,
Wright, P.C. (Eds.), Funology: From Usability to Enjoyment. Kluwer,
Dordrecht, pp. 31–42.
Hassenzahl, M., Burmester, M., Koller, F., 2003. AttrakDiff: Ein
Fragebogen zur Messung wahrgenommener hedonischer und prag-
matischer Qualität. In: Ziegler, J., Szwillus, G. (Eds.), Mensch &
Computer 2003. Interaktion in Bewegung. B.G. Teubner, Stuttgart,
Leipzig, pp. 187–196.
Hassenzahl, M., Kekez, R., Burmester, M., 2002. The importance of a
software’s pragmatic quality depends on usage modes. In: Luczak, H.,
Cakir, A.E., Cakir, G. (Eds.), Proceedings of the 6th international
conference on Work With Display Units (WWDU 2002).
ERGONOMIC Institut für Arbeits- und Sozialforschung, Berlin, pp.
275–276.
Hassenzahl, M., Sandweg, N., 2004. From mental effort to perceived
usability: transforming experiences into summary assessments. In:
Proceedings of the CHI 04 Conference on Human Factors in
Computing Systems. Extended abstracts. ACM, New York, pp.
1283–1286.
Hassenzahl, M., Tractinsky, N., 2006. User experience – a research agenda
[Editorial]. Behavior & Information Technology 25, 91–97.
Iurgel, I., 2004a. From another point of view: art-E-fact. In: Göbel, S.,
Spierling, U., Hoffmann, A., Iurgel, I., Schneider, O., Dechau, J., Feix,
A. (Eds.), Technologies for Interactive Digital Storytelling and
Entertainment: Second International Conference, TIDSE 2004,
Darmstadt, Germany, June 24–26. Springer, Berlinu, Heidelberg, pp.
26–35.
Iurgel, I., 2004b. Narrative dialogues for educational installations. In:
Brna P. (Ed.), Proceedings of the third conference on narrative
and interactive learning environments, NILE 2004, August 10–13, pp.
9–16.
 M. Hassenzahl, D. Ullrich / Interacting with Computers 19 (2007) 429–437
Kahneman, D., 1999. Objective happiness. In: Kahneman, D., Diener, E.,
Schwarz, N. (Eds.), Well-being: The Foundations of Hedonic Quality.
Sage, New York, pp. 3–25.
Kardes, F.R., Posavac, S.S., Cronley, M.L., 2004. Consumer inference: a
review of processes, bases, and judgment contexts. Journal of
Consumer Research 14, 230–256.
Orne, M.T., 1969. Demand characteristics and the concept of quasi-
controls. In: Rosenthal, R., Rosnow, R.L. (Eds.), Artifact in Behav-
ioral Research. Academic Press, New York, pp. 143–179.
Pham, M.T., 2004. The logic of feeling. Journal of Consumer Psychology
14, 360–369.
Redelmeier, D., Kahneman, D., 1996. Patients’ memories of painful
medical treatments: real-time and retrospective evaluations of two
minimally invasive procedures. Pain 116, 3–8.
Rubin, J., 1994. Handbook of Usability Testing: How to Plan, Design,
and Conduct Effective Tests. Wiley, New York.
437
Russell, J.A., 2003. Core affect and the psychological construction of
emotion. Psychological Review 110, 145–172.
Schwarz, N., Clore, G.L., 1983. Mood, misattribution, and judgments of
well-being: informative and directive functions of affective states.
Journal of Personality and Social Psychology 45, 513–523.
Spierling, U., 2005. Interactive digital storytelling: towards a hybrid
conceptual approach. Paper presented at DIGRA 2005, Simon Fraser
University, Burnaby, BC, Canada. Available at: <http://www.di-
Downloaded from https://academic.oup.com/iwc/article/19/4/429/704444 by Bibliotheque Cantonale et Universitaire user on 28 September 2021
gra.org/dl/db/06278.24521.pdf/>.
Zhang, P., Li, N., 2004. Love at first sight or sustained effect? The role
of perceived affective quality on user’s cognitive reactions to
information technology. In: Proceedings of the Twenty-Fifth
International Conference on Information Systems (ICIS), pp.
283–295.
Zijlstra, R., 1993. Efficiency in Work Behaviour. A Design Approach for
Modern Tools. Delft University Press, Delft.