See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221248089

The effect of group discussions in usability inspection: A pilot study

Conference Paper · October 2008

DOI: 10.1145/1463160.1463221 · Source: DBLP

CITATIONS READS
7 58

1 author:

Asbjørn Følstad
SINTEF
126 PUBLICATIONS   4,336 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Social Health Bots View project

ETAPAS View project

All content following this page was uploaded by Asbjørn Følstad on 28 May 2015.

The user has requested enhancement of the downloaded file.

 This is an author's accepted manuscript of an article published in Proceedings of the NordiCHI '08, (pp.
467-670). Copyright: ACM. Available online at: http://dl.acm.org/citation.cfm?id=1463221.

The Effect of Group Discussions in Usability Inspection: a

Pilot Study
Asbjørn Følstad
SINTEF ICT
Forskningsveien 1
0373 Oslo, Norway
+47 22067515
asf@sintef.no

ABSTRACT walkthrough [3], relies heavily on the use of group discussions. In

How do group discussions affect the output of a usability
inspection, as compared to individual evaluators only? This
question was investigated in association with usability inspections
of a music community website. Potential users of the website
participated as evaluators in two usability inspection sessions. All
evaluators made individual predictions prior to group discussions.
Twenty-five percent of the usability issues generated in the group
discussion were new, viz. not predicted by individual evaluators.
Also, the group discussion served to discard or modify the
majority of usability issues predicted by the individual evaluators.
The discarded individual predictions were typically low severity.
Categories and Subject Descriptors
H.5.m [Information interfaces and presentation]: Miscellaneous
General Terms
Human Factors
Keywords
Usability inspection methods, group discussion
1. INTRODUCTION
Individuals and groups perform differently when conducting
analyses and reaching conclusions, something that has been
shown in research within the fields of psychology and
organization studies. Given this difference, it is interesting to
investigate the effect of group discussions in usability inspections.
Usability inspection methods (UIM) vary greatly regarding the
use of group discussions. At one extreme, the heuristic evaluation
is constructed to support the individual evaluator to such a degree
that in the case of multiple evaluators, communication between
evaluators is discouraged until all evaluations have been
completed [1]. In contrast, UIMs involving user representatives,
such as the pluralistic walkthrough [2] and the group-based expert
between these extremes, the cognitive walkthrough can be
performed either individually or in groups [4].
This paper presents a pilot study of the possible effect of group
discussions in usability inspections. The paper makes an
important contribution by providing initial insight in this effect;
insight that is important both as background knowledge to
practitioners and as input to future UIM development.
2. PREVIOUS RESEARCH
In his review of the empirical literature comparing the problem
solving and decision making performance of groups versus
individuals, Hill [5] reported that group performance is generally
superior to the average individual performance, but inferior to the
best individual in a statistical aggregate. Group productivity
seems to increase when members can correct each other and pool
information.
Miner [6], building on the work of Hill, investigated the effect of
the decision process sequence on the performances of groups vs.
individuals. Miner concluded tentatively that when individual
decisions precede the decision of a group, the group outperforms
the best individual decision. Groups without previous individual
decision making were outperformed by groups whose individuals
had made decisions for themselves prior to entering the group.
Related HCI work includes studies by Nielsen and Sears,
indicating that the pooling of results from several evaluators
increases the number of usability problems identified [1] and the
evaluation’s thoroughness [7], as compared to the results of a
single evaluator. These findings indicate that the individuals of a
group of evaluators most likely will contribute with heterogeneous
input; possibly complementing each other in the discussion.
Similarly, a study on the evaluator effect by Hertzum et al. [8]
indicates low overlap between individual evaluators’ usability
problems. In this last study, when the evaluators discussed their
individual findings they concluded that they in general were in
agreement despite their low overlap of identified problems.

3. RESEARCH QUESTION AND

Permission to make digital or hard copies of all or part of this work for HYPOTHESES
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
The research question of the present study was:
copies bear this notice and the full citation on the first page. To copy Which effect will the use of group discussions have on the output
otherwise, or republish, to post on servers or to redistribute to lists, of a usability inspection, as compared to the pooled output of the
requires prior specific permission and/or a fee.
individual evaluators alone?
NordiCHI 2008: Using Bridges, 18-22 October, Lund, Sweden
Copyright 2008 ACM ISBN 978-1-59593-704-9. $5.00
Possible effects of using evaluation groups were formulated as
three hypotheses:
H1: The group discussion will lead to the prediction of new
usability issues.
H2: The group discussion will lead to the discarding or
modification of usability issues predicted by the individual
evaluators.
H3: The group discussion will lead to changes in the severity
ratings of the usability issues predicted by the individual
evaluators.
4. METHOD
The pilot study was conducted as an empirical comparison of the
pooled output of individual evaluators and the output of
subsequent group discussions in a given case.
The case was a usability inspection of a music community
website, Urørt (www.nrk.no/urort), used by unsigned Norwegian
artists to share their demo material with the general public. Urørt
is run by the Norwegian state broadcaster NRK, contains the
music of ~21.000 artists, and is visited by ~12.000 unique users
each day.
The inspection method was the group-based expert walkthrough
[3], a UIM particularly developed to allow non-usability experts
to participate as evaluators. Group-based expert walkthrough is
structured as a series of individual note taking and plenary
discussions, where the evaluators decide individually on usability
issues prior to plenary discussions.
Two evaluation sessions were conducted, with 7 participants in
each. Each session lasted 2.5 hours. The sessions were led by a
usability professional (the author of the present paper). In addition
one secretary and one observer participated. Totally 12 task-
scenarios were included in the two evaluations, targeting
navigation mechanisms, functionality for playing audio content,
and profile page interaction. In one of the sessions 9 task-
scenarios were walked through, in the other 10. For each task-
scenario the test leader first presented the steps required to solve
Table 1: Examples of match between individually and plenary recorded items
Match Individual record Corresponding plenary record
"Link to individual genres"
Weak
match
"Should be placed in a nearby container"
"Why include artists you cannot listen to?"
Fair
match "The house icon is usually front page, slightly
misguiding, but ok if it is there. Somewhat discreet!"
"Name of the song or the artist? Doubled up?"
Good
match "The player opens in a separate window. Would prefer a
player embedded in the artist’s page"
the task, following which the evaluators were required to make
individual notes on predicted usability issues (problems or design
suggestions). Discussions were not allowed during the individual
note taking. Upon completion of the individual notes, the
participants’ individual issues were presented and discussed in the
group. The group decided on whether or not the issue should be
sustained and (if sustained) how it should be formulated. The
discussions were structured by the test leader, but the input to the
discussions was to come from the evaluators only. For each of the
steps of the task scenarios, high-severity issues were presented
and discussed prior to low-severity issues.
The walkthroughs resulted in 14 individual records of predicted
usability issues, as well as two plenary records – one from each
session. All issues, both individually recorded and plenary
decided, were required to be associated with a severity rating. The
following severity ratings were employed: Cosmetic (minor
obstacles or sources of irritation), Serious (major obstacles of
sources of irritation), Critical (insurmountable obstacles or
sources of irritation).
The evaluators were potential, but not actual, users of the website.
All evaluators reported music to be of high importance to them,
and all but three reported to listen to music online weekly or
more. All but four had never used the Urørt website at all. All
used Internet on a daily basis, and all but five also used online
communities on a daily basis. Eight were males and 6 females.
Mean age was 22 years (min. 18, max 24).
When the evaluation sessions were finished, the individual
records were matched with their associated plenary record. For
each item of an individual record, one analyst (the author of this
paper), judged whether or not it matched any item of the
associated plenary record. All items were reported in a format
clearly identifying the task-scenario context, something that
facilitated the matching process. The strength of each match was
rated according to three categories: Weak match (major changes
made to the individually recorded item), Fair match (some
changes), Good match (minor or no changes)
Examples of individual and plenary notes associated with the
three match categories are given in Table 1.
"Not a good idea to mix genres in the recommendation section if
this section only reflects parts of the available genre".
"Wish more information about the artist available in the same
window as the player"
"Annoying with songs in the list that you cannot listen to"
"Difficult to understand the house-icon of the player leading to the
artist's own page. Interpreted as link to the Urørt main page."
"Difficult to differentiate between song name and artist name"
"Annoying that the player opens in a separate window. Wish the
player to be embedded at the artist’s own page, as MySpace"
After the matching, the following analyses were conducted:
New usability issues predicted by the group (H1): Plenary items
not matching any individual’s items were regarded as new;
plenary items matching individual’s items were regarded as
sustained. The proportion of new items to the totality of plenary
items was calculated. The severity ratings of new vs. sustained
items were compared.
The groups’ discarding or modification of usability issues
predicted by individuals (H2): Individual items not matching
any plenary items were regarded as discarded; individual items
associated with the categories Weak match and Fair match were
regarded as modified. The proportions and severity ratings of
discarded, modified and unchanged items were calculated.
Changes in severity rating between individuals and group (H3):
The severity rating of each sustained plenary item was compared
with the average severity rating of its associated items recorded
by individual evaluators. Average severity ratings were
calculated upon assigning the following values to each severity
level: Cosmetic=1, Serious=2, Critical=3.
5. RESULTS
The individual evaluators’ notes contained a total of 175 items
(an additional four items were discarded due to interpretational
issues). The mean number of items for each individual evaluator
was 12.5 (SD=5.2). Thirty-six percent of the individually
recorded items were classified as cosmetic, 35 percent serious
and 18 percent critical. Eleven percent of the individual
evaluators’ items were not rated for severity.
The plenary discussions resulted in 76 items. Thirteen percent of
the plenary recorded items were classified as cosmetic, 38
percent serious and 21 percent critical. Twenty eight percent of
the plenary recorded items were not rated for severity.
Results of relevance to H1: Twenty-five percent of the plenary
recorded items were not found to match any individually
recorded item, and thus regarded as new. The severity ratings of
new vs. sustained items were compared through a Fischer Exact
test. Items not rated for severity were excluded from the test. No
significant difference was found (n=55; p(two tailed)=0.19).
Results of relevance to H2: The distribution of individually
recorded items as discarded, modified, or unchanged is
presented in Table 2.
Table 2: Distribution of individually recorded items
according to match with plenary recorded items
Match Count % strong impact on the items identified by individual evaluators,
No match found (discarded)
43 25
Weak or fair match (modified)
66 37
Good match (unchanged) 66 38
Sum 175 100
The severity ratings of discarded vs. sustained (modified and
unchanged) items were compared through a Chi-square test.
Items not rated for severity were excluded from the test. A
significant difference was found (n=153; X2(2)=15.9; p(two
tailed)=0.00). Distribution details are presented in Table 3.
Table 3: Distribution of discarded vs. sustained individually
recorded items, across severity categories.
Severity Modified or
Discarded items
(individual unchanged items
ratings) Count % Count %
Cosmetic 25 58 36 27
Serious 9 21 52 39
Critical 3 7 28 21
Not rated 6 14 16 12
Sum 43 100 132 100
Results of relevance to H3: Forty-four of the plenary items had
been rated both for severity and associated with one or more
individually recorded items rated for severity. Of these plenary
items, 21 had a severity rating identical to the average severity
score of their individually recorded counterparts, 16 had higher
severity ratings and 7 had lower. The severity ratings of plenary
items vs. individual’s items were compared through a Wilcoxon
signed rank test. No significant difference between the groups
was found (Z=1.40; p(two tailed)=0.16).
6. DISCUSSION
The discussion is structured according to the three research
hypotheses and is concluded by suggestions for refined
hypotheses and further work.
6.1 New usability issues
One fourth of the items of the plenary records were interpreted
as new, resulting from the group discussions. These new items
were not found to differ in severity from the sustained items,
implying that the usability issues generated in the group
discussions were judged by the evaluators to be of equal
important to the issues brought into the discussion by one or
more of the group’s individuals.
H1 is sustained. Group discussions indeed seem to enable the
identification of new usability issues. However, the great
majority of the issues already being identified by the individuals
of the groups, the importance of the group discussions as such
seemed to be a fairly modest source of new issues.
6.2 Discarded or modified usability issues
Twenty-five percent of the individually recorded items were
interpreted as discarded and an additional 37 percent as
modified in the group discussions. The group discussions had
leaving less than half these items unchanged.
Consequently, even though a significant aspect of group
discussions may be to identify new usability issues, more
important is their contribution to qualify the set of usability
issues already identified by the individual evaluators. This effect
of the group discussions is in line with previous research on
group vs. individual performance by Hill [5] and Miner [6].
Such qualification of individual items through group discussions
may be highly valuable. The modification of usability issues
allows the group of evaluators itself to re-focus identified issues,
instead of this being done by a test-leader when analyzing
 individual evaluators’ contributions. Also, the modified usability
issues will most likely represent the best judgment of several
evaluators rather than one.
It is interesting to note that when the group discussions resulted
in the discarding of items, these items were given significantly
lower severity ratings by the individual evaluators than the
sustained items. Thus it may well be that a characteristic of the
group discussions is to remove low-importance predictions,
something that will be important in practical development
contexts with limited resources for redesign.
H2 is sustained, and the results seem to provide background for
a refinement of the hypotheses where the group discussions will
impact the majority of individually identified items.
6.3 Changed severity ratings?
The comparison of plenary recorded items’ severity rating with
the average severity score of the individually recorded items
indicated a tendency towards increased severity ratings in
plenum as opposed to by individuals. This tendency was,
however, not found to be statistically significant, reflecting a
high proportion of ties (situations where the plenary recorded
item and the severity score of the associated individually
recorded items were identical).
It may be possible that the tendency towards increased severity
ratings in consequence of group discussions corresponds to a
general effect. The present case, however, seem to indicate that
such an effect may not be very great.
6.4 Refined hypotheses
The current pilot study allows, in spite of its weaknesses
regarding validity, a refinement of the current hypotheses and
inspiration for future work. The following refinements of the
hypotheses are suggested:
H1’: The group discussion will lead to the prediction of new
usability issues, however fewer than the sustained usability
issues identified by individual evaluators
H2’: The group discussion will lead to the discarding or
modification of the majority of usability issues predicted by
individual evaluators
H3’: The group discussion will be associated with a minor
increase in the severity ratings of the predicted usability issues
as compared to severity ratings provided by individual
evaluators.
6.5 Limitations and future work
This study has several limitations, which is in line with the study
being a pilot. Future studies set up to investigate the refined
hypotheses resulting from this pilot, should in particular be
considered the following in order to improve validity:
• Include a larger number of cases and evaluator types
(external validity)
• Include objects of evaluation from different application
areas (construct validity)
• Include qualitative analyses of the group discussion
(construct validity)
• Use multiple analysts and/or blind matching of individually
and plenary recorded items (internal validity)
View publication stats
In this pilot study, possible effects of group discussions on the
thoroughness, validity, and downstream utility of usability
inspections were not investigated. Such effects on the
performance of usability inspection would make highly relevant
objectives for future work.
Other relevant future work may be to investigate whether or not
the effect of group discussions found in the present study
depends on the particular group process of group-based expert
walkthrough. Relevant other group processes could be group
discussions associated with adaptations of heuristic evaluation
or cognitive walkthrough, or possibly with groups not organized
by a test-leader.
It is the hope of the author that the present study serves as an
inspiration within the HCI community to continue research on
the effect of group discussions in usability inspections. Such
research may in time change the way we conduct usability
inspections.
7. ACKNOWLEDGEMENTS
The reported study was conducted as part of the research project
RECORD, supported by the Norwegian Research Council’s
VERDIKT program.
8. REFERENCES
[1] Nielsen, J. 1992. Finding usability problems through
heuristic evaluation. In Proceedings of the SIGCHI
conference on Human factors in computing systems
(Monterey, California, 1992). CHI 1992. ACM Press, New
York, NY, 373-380.
[2] Bias, R. 1994. The Pluralistic Walkthrough: Coordinated
Empathies. In J. Nielsen, R. L. Mack, Eds. Usability
Inspection Methods, New York, NY, John Wiley & Sons,
63-76.
[3] Følstad, A. 2007 Group-based Expert Walkthrough. In D.
Scapin, E. L.-C. Law, Eds. R3UEMs: Review, Report and
Refine Usability Evaluation Methods. Proceedings of the
3rd. COST294-MAUSE International Workshop, 58-60.
[4] Wharton, C., Rieman, J., Lewis, C., and Polson, P. 1994.
The Cognitive Walkthrough: A Practitioners Guide. In J.
Nielsen, R. L. Mack, Eds. Usability Inspection Methods,
New York, NY, John Wiley & Sons, 105-140.
[5] Hill, G. H. 1982 Group Versus Individual Performance:
Are N + 1 Heads Better Than One? Psychological Bulletin
91(3), 517-539.
[6] Miner, F. C. 1984. Group versus Individual Decision
Making: An Investigation of Performance Measures,
Decision Strategies, and Process Losses/Gains.
Organizational Behavior and Human Performance 33, 112-
124.
[7] Sears, A. 1997. Heuristic walkthroughs: Finding the
problems without the noise. International Journal of
Human-Computer Interaction 9(3), 213-234.
[8] Hertzum, M., Jacobsen, N.E., Molich, R. 2002. Usability
Inspections by Groups of Specialists: Perceived Agreement
in Spite of Disparate Observations. In Extended Abstracts
of the ACM CHI 2002 Conference (Minneapolis, MN,
April 20-25, 2002), ACM Press, New York, NY, 662-663.