Gender in End-User Software Engineering
Margaret Burnett*, Susan Wiedenbeck†, Valentina Grigoreanu*,
Neeraja Subrahmaniyan*, Laura Beckwith*, Cory Kissinger*
*Oregon State University
Corvallis, OR, USA
{burnett,grigorev,subrahmn,beckwith,ckissin}@eecs.orst.edu
ABSTRACT
In this paper, we describe research that reports gender differences
in usage of software engineering tools by end-user programmers.
We connect these findings with possible explanations based on
theories from other disciplines, and then add to that our recent
results that these differences go deeper than software engineering
tool usage to software engineering strategies. We enumerate the
strategies that work better for males and the ones that work better
for females, and discuss implications and possible directions for
follow-up.
Categories and Subject Descriptors
D.2.5 [Software Engineering]: Testing and Debugging; H.1.2
[Information Systems]: User/Machine Systems—Human factors;
H.4.1 [Information Systems Applications]: Office Automation—
Spreadsheets
General Terms
Human Factors, Reliability
Keywords
Gender, debugging, end-user programming, end-user software
engineering, tinkering, self-efficacy, strategy, Surprise-Explain-
Reward.
1. INTRODUCTION
A goal of our research is to support people who engage in end-
user software engineering tasks, such as testing and debugging
spreadsheet formulas. For example, we have developed features
that can be seamlessly blended into spreadsheet software to
encourage and assist end users in systematically testing and
debugging spreadsheets [7]. In the course of this research we
began to notice that females’ behavior with these testing and
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
WEUSE IV’08, May 12, 2008, Leipzig, Germany.
Copyright 2008 ACM 978-1-60558-034-0/08/05...$5.00.
21
†Drexel University
Philadelphia, PA
Susan.Wiedenbeck@cis.drexel.edu
debugging tools seemed to be different than males’ behavior. This
led us to conduct a series of studies investigating whether there
were indeed gender differences pertinent to the design of these
end-user software engineering tools.
Our results showed consistently that male and female end-user
programmers did indeed use debugging tools differently. In this
paper, we first offer possible explanations for these phenomena
from theories in the areas of information processing and problem
solving, demonstrating how these differences should indeed affect
males’ and females’ behaviors during end-user debugging.
Second, we describe our most recent empirical results on gender
in end-user software engineering, focusing on two lines of
research: (1) what strategies males and females employ in
debugging and (2) how two variants of just-in-time explanations
of debugging strategy impact males’ and females’ debugging.
2. BACKGROUND
Areas such as psychology and marketing have identified ways in
which males and females differ, and these differences have been
linked to behaviors in software-based problem-solving tasks such
as end-user debugging [2].
In psychology, Bandura defines the construct of self-efficacy as
an individual’s judgment of his or her ability to carry out a
specific action and thus to attain a desired performance outcome
[1]. Individuals who have low self-efficacy tend to exhibit lower
use of cognitive strategies, less persistence, and lower effort
overall than individuals who have high self-efficacy. Indeed,
studies of learning computer applications showed that females had
lower self-efficacy than males [8, 17], but they did not go on to tie
the effects of self-efficacy to performance outcomes.
An empirical study that our group carried out on the effects of
self-efficacy in end-user debugging did go on to show
downstream effects of self-efficacy [3]. The environment used
was WYSIWYT (“What You See Is What You Test”) which
provides visual debugging tools [7]. Participants in the study,
males and females, were each given a research spreadsheet with
enhanced features to aid debugging. Their self-efficacy was
measured before doing the debugging tasks. As in the studies
above, females had lower self-efficacy than males. But more
importantly, their self-efficacy was tied to their performance.
Females’ level of self-efficacy predicted their final percent
testedness of the spreadsheet—low self-efficacy was associated
with low feature usage, whereas males’ performance was not
predicted by self-efficacy, suggesting that self-efficacy is much
more important in females’ problem solving than males’. Follow-
on studies sometimes found females’ self-efficacy to be lower
than males and sometimes did not—but they consistently found
this phenomenon of a significant tie between females’ self-
efficacy and their success, but no such tie for males [4].
Research in motivation has shown that females perceive higher
risks than males do in many situations [10], including intellectual
risks [9]. Blackwell’s Attention Investment Model [6] describes
risks and benefits in problem solving using programming by end
users who are not necessarily highly skilled. In the model, an
individual considers the perceived benefit of programming and its
expected payoff versus the perceived risk and cost if the
programming fails. If females are risk-averse they may perceive
the risks and cost to greatly outweigh the benefits of
programming. Blackwell’s model applies not only to
programming, but also to other software-based problem-solving
tasks, especially those requiring use of (perceived) sophisticated
or time-intensive devices in the software environment.
In our study above, we did find evidence of risk-averse behaviors
by females. As we have mentioned, several features were
provided to help participants debug spreadsheets. Females showed
significantly lower acceptance of these debugging features. They
were willing to edit formulas, a debugging feature they were
familiar with, but were less willing than males to initially try out
the new features. The females were also less willing to adopt new
features, that is, to intellectually engage with the features in
repeated usage. In a post-questionnaire, females said that they did
not use a new feature because they thought it would take too long
to learn, suggesting that risk was an issue with them and is a
barrier to adopting new features that could improve their
debugging. Here again self-efficacy arises; although the females
said they thought it would take them too long to learn the features,
by the end of the task, our measures of feature comprehension
showed no differences in how well males versus females actually
understood the features.
A partial replication of the study above, with real-world end-user
developers using Excel on a real-world spreadsheet, showed the
findings to be robust. It replicated the finding that females’ self-
efficacy predicts effectiveness, while males’ self-efficacy does
not. Furthermore, females in the Excel study again focused on the
most familiar features, especially value edits in this study,
suggesting that they may have avoided more complicated features
(or at least those features that were perceived to be
complicated)—something that did not occur among the males.
We then carried out a follow-on empirical study to understand
male and female end-user programmers’ exploration and self-
learning of spreadsheet testing and debugging features. In this
study we used two environments, one similar to the prior study
(called the low-cost environment) and a second environment that
was also similar but designed to provide greater support (called
the high-support environment).
Participants engaged in tinkering (or playful experimentation) in
exploring and using new features. Educational literature reveals
that hands-on, self-guided tinkering can benefit learners [14].
However, the educational literature also reports that males have a
greater propensity to tinker than do females [11].
In the study, males tinkered a great deal, but across both
environments their tinkering did not predict effective testing nor
22
number of bugs fixed. Males tinkered highly, even excessively, in
the low-cost interface, which makes tinkering easy to do. They
tended to tinker without pausing to reflect, and this may have
reduced both the educational benefit of tinkering and
correspondingly their effectiveness in debugging [14]. In contrast
to males, females tinkered equally in both interfaces, and their
tinkering was predictive of testing effectiveness and increased
bugs fixed. Females paused during tinkering more than males did.
Tinkering in a moderate and “pause-ful” manner appears to have
helped the females learn about the features, leading to effective
outcomes. Nevertheless, self-efficacy remained an important
predictor of effectiveness in females’ debugging, interacting with
the benefits gained in tinkering. For example, in some cases in the
high-support environment, females’ tinkering actually interfered
with their self-efficacy.
3. CURRENT AND FUTURE RESEARCH
DIRECTIONS
3.1 Strategies
Because of the tool usage and behavior differences we had found
in end-user debugging, we wondered whether the differences went
deeper than mere behaviors, down to their strategies. Thus, we
decided to investigate what debugging strategies end-user
programmers were actually trying to follow, and whether there
were gender differences in those strategies.
Research in the area of marketing also provides reasons to think
that there may be gender differences in end-user debugging
strategies. The Selectivity Hypothesis [13] proposes that males
and females behave differently in decision making and problem
solving. Females tend to process information in a comprehensive
way, examining all the available cues and making elaborative
inferences in order to make a decision. Furthermore, they practice
comprehensive processing whether the problem is simple or
complex. Males, on the other hand, avoid comprehensive
information processing, using heuristics to make decisions and
only falling back on comprehensive processing if a complex task
requires it. If the Selectivity Hypothesis holds across multiple
areas involving problem solving and decision making, female end
users may be most effective using debugging strategies that make
good use of comprehensive processing.
There are many potential strategies for debugging spreadsheets.
For example, one might begin by testing, and then use a data flow
strategy to narrow down the possible cause of incorrect output.
Alternatively, one might inspect the code looking for logic flaws.
To discover the strategies males and females attempted to use in
debugging, we asked them via an open-ended question what
strategies they used. We then combined these verbal descriptions
of strategies with behavioral evidence of these strategies, to
determine whether males and females actually used different
strategies in debugging and whether those strategy choices led to
success.
Their responses revealed eight strategies: testing, dataflow, code
inspection, specification checking, color following, to-do listing,
fixing formulas, and spatial. Three strategies stood out: testing,
dataflow, and code inspection.
The behavioral evidence showed that testing values to verify
cells’ correctness was a successful strategy for males. In
particular, the males who had high success in debugging used
testing more than the less successful males; likewise, successful
male debuggers used testing more than successful female
debuggers. Males also used dataflow strategies more than did
females, and furthermore males using dataflow were more
successful at the end of the experiment than were females who
used dataflow.
By contrast, code inspection was more associated with females;
successful females had more instances of code inspection than
males and more total formulas displayed in all instances.
Referring back to the Selectivity Hypothesis, it may be that
females using code inspection were practicing comprehensive
processing, investigating many formulas in detail. (Males, in
contrast, may have been successful because strictly following
dataflow arrows through the spreadsheet minimizes the amount of
information processing required.) Other strategies that were
successful for females but not for males were specification
following in which users verified formulas via comparison to a
specification document, as well as to-do listing (marking cells to
track code inspection progress).
Overall, of the eight strategies the end-user participants described,
there were gender differences for seven of them! A main result of
this study was that debugging strategies that worked well for
males did not work well for females. A disadvantage for female
debuggers was that, for the most part, their preferred strategies of
code inspection, specification following, and to-do listing, are not
well supported by features in end-user environments. Future work
may assess whether new design features can provide better
support for these strategies.
3.2 Strategy Explanations
We have also been exploring how to support the end-user
programmers who struggle to find a suitable strategy, because the
above study also revealed that not everyone used reasonable
strategies to track down the bugs. In particular, we would like to
entice them into strategies that have a chance of working well for
them.
The WYSIWYT environment in which we prototype our
approaches provides a motivational strategy, called Surprise-
Explain-Reward, which attempts to entice end-user programmers
to use end-user software engineering features for systematic
testing and debugging [18]. The first step in the strategy is to
(gently) surprise the user with something unexpected in the
environment, such as a red colored cell border. Second, if the user
is curious about it, he or she can seek an explanation via tool tips
that pop up when the user hovers over the red cell border. Finally,
if the user reads and carries out what is advised in the tool tip,
rewards may occur; for example, the feature might help them find
the bug. While the surprises and rewards are fairly well
developed, there is still much to do in determining what
explanations users want and how they can most effectively be
conveyed.
We have conducted an initial qualitative empirical study to
determine what types of explanations users need in debugging
[12]. These explanations fell into five groups. One of the most
prominent needs elicited from the study participants was
explanations of how to proceed in debugging, e.g., what would be
a suitable strategy or how to accomplish a particular goal. This
type of information gap was by far greater than explanations of
new features.
Building on these new findings on information gaps, we carried
out a qualitative empirical study focusing on explaining ways to
23
approach debugging strategies, rather than explaining features.
Six such explanations were developed: how to find errors, how to
fix errors, how I can test my spreadsheet, why should I change
values, what is a good overall strategy, and am I doing it right.
One important question is whether participants would take the
time to learn via the explanations—the Attention Investment
Model makes clear that individuals assess the potential benefits
and costs in order to make a decision.
Tool tips in WYSIWYT addressed part of these explanations but
were limited to a short text, and did not seem to be a viable way to
explain strategies. We therefore chose two new two vehicles for
short strategy explanations: video explanation snippets (1-2.5
minutes) and hyperlinked textual explanations. The videos
showed two people (one male and one female) problem solving
and discussing one of the six strategies, while demonstrating how
to carry them out on a spreadsheet. The female took on the role of
the confused or questioning student, who in the end of the video
was always successful. This was done because one way of
increasing self-efficacy is to view another person like oneself
struggle in a task and ultimately succeed [1]. The wording of the
text version was identical to the video except that there was no
sample spreadsheet to which the reader could refer. The
participants were 7 males and 3 females. When they needed an
explanation during the course of debugging, they had the option
of using either the text version or the video.
The results showed that the explanations improved the
participants’ choices, allowing them to close information gaps that
hindered their debugging. In addition, females reported an
increase in their confidence due to the explanations, but no male
suggested this. In this small group of participants the choices of
media varied, suggesting both presentation choices should be
provided. The findings also provided recommendations for
changes to future versions of the explanations: some participants
lacked the motivation to read or view the explanations and some
misinterpreted the explanations.
4. IMPLICATIONS AND CONCLUSIONS
The implications of our findings so far are quite clear: the features
in supposedly gender-neutral end-user software environments are
not gender-neutral after all. Males’ and females’ use of and
benefits from these features vary greatly.
Currently, we are designing a larger and more definitive empirical
study of strategy explanations and gender. We are also
investigating whether small changes to the attributes of features
may remove some of the barriers to males’ or females’ success
with and willingness to use those features.
We point out that there are no “typical” males or “typical”
females. For example, many males use comprehensive
information processing, and many females use heuristic
processing. Therefore, we are not advocating separate versions of
software for each gender. Rather, we advocate adjusting features
and offering flexible options so that any end-user programmer
whose cognitive or problem-solving style does not fall into the
patterns preferred by the developer of that software can still be
effective in their software development efforts.
5. ACKNOWLEDGMENTS
This work was supported in part by Microsoft Research and by
NSF CNS-0420533, ITR-0325273 and CCR-0324844.
6. REFERENCES
[1] Bandura, A. Self-efficacy: Toward a unifying theory of
behavioral change. Psychological Review 8, 2 (1977), 191-
215.
[2] Beckwith, L. and Burnett M. Gender: An important factor in
end-user programming environments? In Proc. IEEE
Symposium on Visual Languages and Human-Centric
Computing (2004), 107-114.
[3] Beckwith, L. Burnett, M., Wiedenbeck, S., Cook, C., Sorte,
S., and Hastings, M. Effectiveness of end-user debugging
software features: Are there gender issues? In Proc. CHI
2005, ACM Press (2005), 869-878.
[4] Beckwith, L., Inman, D., Rector, K., Burnett, M. On to the
real world: Gender and self-efficacy in Excel, In Proc.
VLHCC, IEEE (2007).
[5] Beckwith, L. Kissinger, C., Burnett, M., Wiedenbeck, S.,
Lawrance, J., Blackwell, A., and Cook, C. Tinkering and
gender in end-user programmers’ debugging, In Proc. CHI
2006, ACM Press (2006), 231-240.
[6] Blackwell, A. First steps in programming: a rationale for
attention investment models. In Proc. IEEE Human-Centric
Computing Languages and Environments (2002), 2-10.
[7] Burnett, M., Cook, C. and Rothermel G. End-user software
engineering. Communications of the ACM 47, 9 (2004), 53-
58.
[8] Busch, T. Gender differences in self-efficacy and attitudes
toward computers. Journal of Educational Computing
Research 12, 2 (1995), 147-158.
[9] Byrnes, J. P., Miller, D. C. and Schafer W. D. Gender
differences in risk taking: A meta-analysis. Psychological
Bulletin 125, (1999), 367-383.
[10] Finucane, M., Slovic, P., Merz., C-K., Flynn, J. and
Satterfield, T. Gender, race and perceived risk: the white
male effect. Health, Risk and Society 2, 2 (2000), 159-172.
24
[11] Jones, M. G., Brader-Araje, L., Carboni, L. W., Carter, G.,
Rua, M. J., Banilower, E. and Hatch, H. Tool time: Gender
and students’ use of tools, control, and authority. Journal of
Research in Science Teaching 37, 8 (2000), 760-783.
[12] Kissinger, C., Burnett, M., Stumpf, S., Subrahmaniyan, N.,
Beckwith, L., Yang, S., and Rosson, M. B. Supporting end-
user debugging: What do users want to know? In Proc.
Advanced Visual Interfaces, ACM Press (2006), 135-142.
[13] Meyers-Levy, J. Gender differences in information
processing: A selectivity interpretation. In P. Caffarata & A.
Tybout (Eds.), Cognitive and Affective Responses to
Adverrtising. Lexington, MA, Lexington Books, 1987.
[14] Rowe, M.B. Teaching Science as Continuous Inquiry: A
Basic (2nd ed.). McGraw-Hill, New York, NY 1978.
[15] Subrahmaniyan N., Beckwith, L., Grigoreanu, V., Burnett,
M., Wiedenbeck, S., Narayanan, V., Bucht K., Drummond,
R., Fern, X. Testing vs. Code Inspection vs. ... What Else?
Male and Female End Users’ Debugging Strategies. In Proc.
CHI 2008, ACM Press (to appear).
[16] Subrahmaniyan N., Kissinger, C., Rector, K., Inman, D.,
Kaplan, J., Beckwith, L., and Burnett, M. Explaining
debugging strategies to end-user programmers. In Proc.
IEEE Symposium on Visual Languages and Human-Centric
Computing (2007), 127-134.
[17] Torkzadeh, G. and Koufteros, X. Factorial validity of a
computer self-efficacy scale and the impact of computer
training. Educational and Psychological Measurement 54, 3
(1994), 813-821.
[18] Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn,
L., Cook, C., Durham, M. and Rothermel, G. Harnessing
curiosity to increase correctness in end-user programming. In
Proc. CHI 2003, ACM Press (2003), 305–312.