Information & Management 37 (2000) 1±12

Research

Using a structured design approach to reduce risks

in end user spreadsheet development
Diane Janvrina,*, Joline Morrisonb
a
Department of Accounting, University of Iowa, Iowa City, IA 52242, USA
b
Department of MIS, University of Wisconsin, Eau Claire Eau Claire, WI 54702, USA

Received 31 July 1998; accepted 24 May 1999

Abstract

Computations performed using end-user developed spreadsheets have resulted in serious errors and represent a major
control risk to organizations. Literature suggests that factors contributing to spreadsheet errors include developer inexperience,
poor design approaches, application types, problem complexity, time pressure, and presence or absence of review procedures.
We explore the impact of using a structured design approach for spreadsheet development. We used two ®eld experiments and
found that subjects using the design approach showed a signi®cant reduction in the number of `linking errors,' i.e., mistakes in
creating links between values that must connect one area of the spreadsheet to another or from one worksheet to another in a
common workbook. Our results provide evidence that design approaches that explicitly identify potential error factors may
improve end-user application reliability. We also observed that factors such as gender, application expertise, and workgroup
con®guration also in¯uenced spreadsheet error rates. # 2000 Elsevier Science B.V. All rights reserved.

Keywords: Risks in end user development; Spreadsheet errors; Structured design

1. Introduction costing [39]. A large mutual fund incorrectly reported

During the past decade, the use of end-user devel-
oped applications, especially spreadsheets, has grown
exponentially. However, when identifying and evalu-
ating risks within organizations, end-user developed
computer applications are often overlooked. Studies
have revealed that as many as 44% of all end-user
spreadsheets contain at least one error [5]. Such errors
can have catastrophic results. For example, a Florida
based construction company lost US$ 254,000 by
relying on an incorrect spreadsheet analysis for project
*
Corresponding author.
E-mail addresses: diane-janvrin@uiowa.edu (D. Janvrin),
morrisjp@uwec.edu (J. Morrison)
a US$ 0.1 billion loss as a US$ 2.3 billion gain by
placing the wrong sign on a US$ 1.2 billion spread-
sheet input entry [38].
Previous effort suggests that errors in end-user
developed applications may be in¯uenced by several
factors [28,30]. Here we develop a framework to
examine end user risks and speci®cally to investigate
the impact of one risk mitigating factor, a structured
design approach, on spreadsheet error rates and devel-
opment times.
2. Background and model development
Cotterman and Kumar [9] classify end-user com-
puting risks into to three major parts: end-user devel-

0378-7206/00/$ ± see front matter # 2000 Elsevier Science B.V. All rights reserved.
PII: S 0 3 7 8 - 7 2 0 6 ( 9 9 ) 0 0 0 2 9 - 4
2 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12
opment, end-user operation, and end-user control of
information systems. We focus on the domain of end-
user development. To our knowledge, no work has
speci®cally identi®ed and then empirically validated
factors impacting outcomes in end-user developed
applications, such as spreadsheets. Literature suggests
that potential factors include developer attributes,
development approach, developer con®guration, pro-
blem/process characteristics and application domain.
2.1. Developer attributes
Our work focuses on end-users who are both pro-
ducers and consumers of information. Several studies
have indicated that end-users are diverse, and that this
diversity leads to a need for differentiated education,
training, support, software tools and risk assessment
[3,12,14,19,34].
Harrison and Rainer [17] assert that speci®c end-
user attributes include age, gender, computer experi-
ence, computer con®dence, math anxiety, and cogni-
tive style. Curtis et. al. ®eld studies on software
development [10] repeatedly con®rm the importance
for software developers to understand their application
domain. Adelson and Soloway [1] emphasize the
importance of expertise in software design. Frank,
Shamir and Briggs [15] ®nd that PC user knowledge
± i.e., application expertise ± positively impacted
security-related behavior. Further, several researchers
indicate that user training and relevant experience are
important factors e.g., [2,37].
2.2. Development approach
Design activities for end-user developed applica-
tions tend to be unstructured and often non-existent.
Salchenberger notes that end-users tend to develop
and implement models without performing require-
ments analysis or developing a system design. Brown
and Gould observed that their participants spent little
time planning before launching into spreadsheet cod-
ing. Ronen, Palley and Lucas [35] suggest that several
attributes of spreadsheets (i.e., non-professional
developers, shorter development life cycles, modi®ca-
tion ease) tend to preclude a formal spreadsheet
analysis and design process.
Almost all published spreadsheet design appr-
oaches agree on one principle: the importance of
keeping work areas small in order to make errors
easier to ®nd, limit the damaging effects of errors,
and make spreadsheets easier to understand and main-
tain [33]. Markus and Keil [24] note that structured
design methodologies increase the accuracy and
reliability of information systems. Salchenberger
proposes the use of structured design methodologies
in end-user development and discusses their effective-
ness in different situations. Cheney, Mann and Amor-
oso [7] also support the use of a structured design
methodology; they found that the more structured the
tasks of the end-user, the more likely the success of
the application. Ronen, Palley and Lucas describe a
structured spreadsheet design approach based on
modifying conventional data ¯ow diagram symbols
to create spreadsheet ¯ow diagrams. Several practi-
tioner-directed articles encourage the use of structured
design approaches and planning for spreadsheet devel-
opment.
2.3. Developer configuration
Panko and Halverson [29,31] and Nardi and Miller
[26] note that the accuracy of end-user developed
applications might be increased by encouraging devel-
opers to work in teams. However, the former also
suggest that the increased communication and coor-
dination costs of working in groups may outweigh the
bene®ts of reduced error rates.
2.4. Problem/process characteristics
Several studies identify the dif®culties associated
with complex versus non-complex programming pro-
blems e.g., [13,36]. Studies on the effect of time
pressure on programming activities indicate that due
to time pressure, end-user developers may not spend
suf®cient time on problem de®nition and diagnosis
e.g., [2,6]. They also note an absence of formal review
and validation procedures within end-user developed
applications.
2.5. Application domain
While this research covers a wide range of applica-
tion domains, spreadsheet development should be a
central issue, because of the widespread use of end-
user developed spreadsheets in the workplace. Panko
 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12 3

and Halverson appropriately summarize the state of
spreadsheet research as follows:
In the past, spreadsheeting has been the Rodney
Dangerfield [e.g., a not well respected subject]
of computer applications. Although everyone
agrees that it is extremely important, few
researchers have studied it in any depth. Given
its potential for harm as well as for good, this
situation needs to change.
Psychologists Hendry and Green [20] assert that
spreadsheets may simplify programming by suppres-
sing the complexities associated with conceptually
simple operations like adding a list of numbers or
handling input and output. As a result, users tend to
ignore their weaknesses, such as their potential for
making errors or their dif®culties in debugging or
reusing spreadsheets.
Eureka error is identi®ed, everyone can quickly agree
that it exists. However, non-Eureka errors are those
where the mistake cannot be easily demonstrated.
Generally, researchers evaluate the usability of end-
user developed applications with satisfaction survey
tools [14,22]. Salchenberger de®nes maintainability
as the ability to change or update the system as needs
change, and auditability as the ability to determine the
source of data values in the model. Few researchers
have evaluated end-user developed applications for
either maintainability or auditability [32]. We also
propose the inclusion of cost (usually measured in
terms of development time) as another important
outcome.
3. Structured spreadsheet design

2.6. Outcomes Based on our review, we developed an initial

research framework (Fig. 1) that summarizes risk
Salchenberger suggests that factors to use in jud- factors and their relationships to potential outcomes.
ging the quality of end-user developed applications Our focus is to determine the impacts of development
include reliability, ease of use, maintainability, and approach on spreadsheet reliability and cost.
auditability. Reliability can be de®ned as the absence Modern spreadsheet applications enable users to
of logical and mathematical errors. One of the earliest modularize large spreadsheets by creating `work-
studies on spreadsheet outcomes (conducted by
Brown and Gould) ®nd that of 27 spreadsheets created
by experienced spreadsheet users, 44% contained
user-generated errors. In another study, Davies and
Ikin [11] audited a sample of 19 spreadsheets in an
organization and discovered four worksheets con-
tained major errors and 13 had inadequate documen-
tation. Panko and Halverson [29] ask students to
develop a spreadsheet model working individually,
in groups of two, and in groups of four, and observed
total error rates ranging from 53±80%. They also note
that very little time (only a few minutes) was spent in
design activities prior to implementation. They also
note that their participants made several different
types of errors. At the highest level, errors could be
divided into two basic types: oversight and concep-
tual. They de®ned oversight errors as `dumb' ones
such as typographical errors and misreading numbers
on the problem statement. In contrast, conceptual
errors involve domain knowledge or modeling logic
mistakes. Panko and Halverson use Lorge and Solo-
mon's [23] terminology to further divide conceptual
errors into Eureka and non-Eureka errors. Once a Fig. 1. Initial research framework.
4 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12
Fig. 2. Basic modeling symbols.
sheets' (individual spreadsheet work areas) within a
larger `workbook' (set of related worksheets). Users
can link values from one worksheet to another within
the same workbook, or link sheets among multiple
workbooks. This technique readily supports scenario
analysis. For example, all of the schedules leading to a
master budget can be prepared on separate worksheets
within a workbook, and then linked together so that
the impact of a change in one parameter can be quickly
evaluated. However, this feature also leads to the
likelihood of `linking errors.' Data on a source work-
sheet may be linked to the wrong cell on the target
worksheet or a value on a target worksheet that should
have been linked from a previous worksheet is `hard-
coded:' an actual value from a previous scenario is
typed in. In such cases, when the value should change
for subsequent scenarios, they do not and errors result.
Also, values that are unknowingly linked to subse-
quent sheets might be deleted, thus destroying the
integrity of the subsequent worksheet. To support the
task of designing linked worksheets, we build upon the
work of Ronen, Palley and Lucas to develop a design
methodology based on data ¯ow diagram symbols.
Data ¯ow diagrams (DFDs) are documentation
tools used by systems analysts to show organizational
data sources, destinations, processes, and data inputs
Fig. 3. Example design model: master budget for small home fixture manufacturer.
and outputs. Our basic modeling symbols (Fig. 2) are
similar to typical symbols in DFD.
We assumed that explicitly identifying links in the
design stage should reduce linking errors made during
the spreadsheet coding phase. Fig. 3 shows an example
design model. Worksheet pages are created to repre-
sent each of the modules shown on the design model
(e.g., Input Page, Sales Budget, Production Budget,
Pro Forma Income Statement). Inputs from external
sources are entered on a centralized Input Page, and
then ¯ow into the Sales Budget, where they are
transformed into data items needed for calculations
on the other schedules. Using named variables, the
speci®ed links are created. Inputs are easily changed
on the Input Page for scenario analysis.
4. Evaluation studies
Based on the research framework, we designed a
®eld experiment to explore the impacts of the struc-
tured design approach on spreadsheet errors. The
reduced research model guiding the study is shown
in Fig. 4. Harrison and Rainer note a positive relation-
ship between developer con®dence and improved
program outcomes. Other researchers identify a posi-
tive relationship between domain and application
expertise and improved outcomes. Therefore, we
hypothesized that both developer attributes and devel-
opment approach would in¯uence error rate and devel-
opment time.
Structured systems design approaches e.g. [8,16]
have successfully been used in systems development
 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12
Fig. 4. Reduced research model for experimental studies.
for over 25 years. They also help in making reusable
components. This approach can be applied to spread-
sheet design by identifying different worksheets and
their corresponding data in¯ows and out¯ows. Thus,
each individual worksheet effectively becomes a
structured module that takes a de®ned set of inputs
and translates it into a prescribed set of outputs. Since
the worksheet links are explicitly de®ned in the
design, we hypothesized that the result should be a
decreased error rate for linking errors.
4.1. Study #1
4.1.1. Subjects, treatment, and measures
Sixty-one upper- and masters-level accounting and
business administration majors in an accounting infor-
mation systems course at a large public university
were randomly assigned to two groups: treatment
(structured design) and control (ad hoc design). All
subjects were given 1 hour of formal training in
spreadsheet development and told to develop a single
spreadsheet workbook containing a series of linked
worksheets using Microsoft Excel. The assigned pro-
blem contained 10 separate worksheets with a total of
51 linked data values among the worksheets. A paper
template of the worksheets with one complete data set
(from Moriarity and Allen [25]) was given to all
subjects to provide check ®gures and mitigate non-
linking errors. (Only numerical values were speci®ed,
and the subjects still had to develop and correctly
implement the underlying design in order to success-
fully perform scenario analysis.) All subjects were
required to maintain logs recording time spent on
design and development activities. The treatment
5
group was required to submit their spreadsheet design
prior to being assigned a password to provide com-
puter data entry.
A pre-test questionnaire was administered to iden-
tify gender, age, and class level, as well as self-
reported pre-test application expertise (in terms of
both spreadsheets in general as well as the speci®c
spreadsheet application, Microsoft Excel), DFD
development expertise, and design con®dence. Devel-
opment time was determined by combining two mea-
sures: (1) self-reported design logs covering two- or
three-day periods during the experiment; and (2)
system-recorded measures of actual implementation
times. The subjects were given 16 days to complete the
assignment. At the end of that period, questionnaires
were administered to ascertain post-test spreadsheet
and Excel expertise and design and coding con®dence.
Error rates were determined by examining the spread-
sheet data ®les.
4.2. Analysis and results
Due to an unexpected shortage of software and
hardware to complete the assignment, we were forced
to allow subjects in Study #1 to work either indivi-
dually or in self-selected pairs within treatment
groups. In Study #2 all subjects worked alone. Means
and standard deviations for the independent and
dependent variables for all subjects as well as results
sorted by con®guration and treatment group are shown
in Table 1. Analysis was performed on a per-subject
basis, with scores for time and error rate replicated for
each subject in a given dyad. A selection internal
validity threat exists, since individuals were assigned
to the treatment group based on their class section.
Pre-test questionnaire data was used to determine the
impact of the selection problem. The treatment group
was signi®cantly older than the control group
(p 0.03), but no signi®cant differences were noted
for pre-test spreadsheet expertise, Excel expertise, or
DFD expertise.
We used least squares multiple regression to test the
research model components and proposed linkages.
Examination of the correlations among the indepen-
dent variables revealed no evidence of multicollinear-
ity: that is, r  0.8 [4]. Table 2 shows that gender, age
group, class level, and pre-test Excel expertise had no
impact on any of the outcomes, while con®guration
6 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12

Table 1
Means and standard deviations (Study #1)

n By configuration By treatment

Overall Singles Pairs Ad hoc design Struct. design

61 17 44 31 30

Variable Mean SD Mean SD Mean SD Mean SD Mean SD

Gender (1 Male, 2 Female) 1.44 0.50 1.50 0.51 1.43 0.50 1.42 0.50 1.47 0.51
Age group (1 20±25, 2 26±30, 3 31±40, 4 40±50) 1.38 0.78 1.56 0.98 1.30 0.67 1.19 0.54 1.57 0.94
Class level (1 Jr., 2 Sr., 3 Grad) 1.90 0.68 2.11 0.58 1.82 0.69 1.97 0.60 1.83 0.75
Pre-test SS Expertisea 2.49 1.07 2.61 1.20 2.43 1.02 2.68 1.05 2.30 1.09
Pre-test Excel expertisea 2.34 1.14 2.22 1.00 2.41 1.19 2.42 1.23 2.27 1.05
Pre-test DFD expertisea 2.16 0.99 1.89 0.83 2.30 1.02 2.26 1.00 2.07 0.98
Pre-test design confidenceb 3.93 0.73 3.72 0.89 4.02 0.63 3.84 0.78 4.03 0.67
Post-test SS expertisea 3.20 0.94 2.87 0.99 3.32 0.88 3.27 0.92 3.12 0.97
Post-test Excel expertisea 3.12 0.89 2.73 0.88 3.30 0.85 3.19 0.85 3.04 0.93
Post-test design confidenceb 2.55 0.90 2.93 1.03 2.38 0.79 2.31 0.84 2.80 0.91
Post-test coding confidenceb 2.48 0.89 2.40 0.91 2.50 0.88 2.48 0.96 2.48 0.82
Mean number of linking errors/workbookc 4.16 0.08 4.85 0.12 3.89 0.05 4.91 0.09 3.37 0.05
Error% 8.15% 0.08 9.51% 0.12 7.62% 0.05 9.63% 0.09 6.61% 0.05
Time (h) 8.83 4.12 8.05 2.89 9.12 4.51 9.98 6.66 8.33 2.37
a
1 Novice, 5 Expert.
b
1 Very unconfident, 5 Very confident.
c
Total possible links 51.

Table 2
Summary of multiple regressions results (Study #1) N 66

Independent variables Standardized regression coefficients

Post SS Post excel Post. des. Post. coding Error% Time

expertise expertise conf. conf.

Gender (1 Male, 2 Female) ÿ0.04 ÿ0.21 0.00 ÿ0.44 0.01 122.99

Age group (1 20±25, 2 26±30, 3 31±40, 4 41±50) ÿ0.02 0.10 ÿ0.26 ÿ0.20 0.01 ÿ18.79
Class level (1 Jr., 2 Sr., 3 Grad.) ÿ0.04 ÿ0.05 0.20 ÿ0.13 0.00 9.21
Configuration (1 lnd., 2 Pair) 0.47a 0.41 0.64b ÿ0.05 ÿ0.03 ÿ4.22
Treatment group (1 Control, 2 Treatment) 0.00 ÿ0.05 0.48a 0.02 ÿ0.04b ÿ115.70
Pre-test SS expertise 0.35b 0.24a ÿ0.08 ÿ0.06 ÿ0.02 ÿ137.17a
Pre-test Excel expertise ÿ0.02 0.27 0.04 0.01 0.00 8.97
Pre-test DFD experitise 0.09 0.14 ÿ0.09 ÿ0.16 0.01 38.83
Pre-test design confidence ÿ0.22 0.18 0.22 0.38 0.01 15.68
R2 0.40 0.41 0.27 0.25 0.16 0.30
a
p  1.
b
p  0.05.

(singles versus pairs), treatment group (structured
versus ad hoc design), and pre-test spreadsheet exper-
tise all displayed signi®cant correlations with one or
more of the dependent variables.
Following Heise [18], the independent variables
that did not explain a signi®cant variation in the
outcomes were excluded from further analysis. Table
3 shows that the variations in the dependent variables
are explained, to some extent, by the independent
variables in the reduced model. MANOVA analysis
was used to detect potential interactions between the
con®guration and treatment variables on error rate.
 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12
Table 3
Reduced model regression results (Study #1) N 66
Independent variables Standardized regression coefficients
Post SS Post excel
expertise expertise
Configuration (1 lnd., 2 Pair) 0.47a 0.65b
Treatment (1 Control, 2 Design) ÿ0.03 ÿ0.06
Pre. SS Exp. (1 Low, 5 High) 0.46c 0.31b
R2 0.35b 0.27b
a
p  1.
b
p  0.5.
c
p  0.001.
This revealed a weak signi®cant difference (p 10)
between the control to the treatment group, as well
as a weak signi®cant interaction (p 10) among the
con®gurations and the treatment.
4.3. Summary and discussion
Overall, subjects using the structured design tech-
nique exhibited signi®cantly lower error rates: of the
51 possible links in the spreadsheet, the structured
design subjects had an average of 3.57 (7%) linking
errors, while the ad hoc design subjects averaged 4.1
(10%) linking errors. We believe this was because the
design methodology forced the developers to identify
the links between worksheets prior to implementation.
This explicit identi®cation was then carried over into
the implementation. Virtually all of the linking errors
were from omitted links (where a value had been `hard
coded' instead of linked) rather than incorrect linkage.
Possibly subjects were relying heavily on the check
®gures provided for the ®rst scenario instead of exam-
ining the reasonableness of their answers for the
subsequent cases.
Improvement in linking errors was mitigated by
subject con®guration: subjects working alone within
the ad hoc design group averaged 7.44 linking errors
per spreadsheet (14%), while subjects working alone
in the design group and both sets of paired subjects all
averaged approximately 3.8 linking errors per spread-
sheet (7%).
For subjects working alone, the MANOVA analysis
results imply that the use of the design methodology
had a signi®cant impact. Unfortunately, the advantage
of using the structured design methodology was wea-
7
Post. des. Post. coding Error% Time
conf. conf.
ÿ0.61b 0.06 ÿ0.02 59.09
0.45a ÿ0.06 ÿ0.03a ÿ233.13b
ÿ0.13 ÿ0.25b ÿ0.01b ÿ76.28a
0.19b 0.10 0.12a 0.27a
kened when the subjects were working in pairs. Two
potential explanations occurred to us: (1) subjects
working in pairs gained other advantages that offset
the impact of the design approach; or (2) another
external factor induced some subjects to elect to work
in pairs and it allowed them to create spreadsheets
with fewer linking errors.
Our results also indicated that subjects working in
pairs reported higher post-test spreadsheet expertise
but lower design con®dence. The latter result seemed
inconsistent with the prior literature. The signi®cantly
lower overall time investment for the structured design
subjects was unexpected but encouraging. (This result
does not consider the time spent initially learning the
design methodology, however). Finally, as expected,
pre-test spreadsheet expertise was a signi®cant pre-
dictor of post-test spreadsheet expertise, reduced error
rates, and reduced time investments. The results of this
study suggested that developer attributes, con®gura-
tions, and the development approaches are all poten-
tial factors impacting end-user development out-
comes. However, the unexpected results for time
and post-test con®dence as well as the unexpected
con®guration variable induced us to perform a second
study with revised procedures and measures.
4.4. Study #2
To con®rm the results of Study #1 and remove the
possible confound involved in allowing subjects to
work in self-selected pairs, a second study was per-
formed. The problem used was slightly more complex
than the one in Study #1, with 10 different worksheets
and a total of 66 linked cells. All subjects were
8 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12

required to work individually so as to increase sample
size and allow a more isolated investigation of the
impacts of the design methodology and developer
attributes. Error types were expanded to include con-
ceptual and oversight errors as well as linking errors.
A blank template (from [21]) with only a single check
®gure was given. Explicit post-test measures were
introduced to assess design con®dence, coding con-
®dence, and con®dence that the spreadsheet was error-
free. Additionally, a measure was included to explore
the impact of potential domain expertise: subjects who
were accounting majors had additional experience and
coursework in preparing ®nancial reports and other
master budgeting topics than the non-accounting
majors. Subjects in the treatment group were also
asked to rate their post-treatment DFD development
con®dence, and to rate their satisfaction with the
development approach. Speci®cally, they were asked
the degree to which they agreed or disagreed with
statements regarding: (1) whether the DFD design
methodology helped them develop a better spread-
sheet; (2) if using the methodology made the devel-
opment effort take longer; and (3) whether they would
use the methodology the next time they developed a
complex spreadsheet.
4.5. Analysis and results
Table 4 shows overall means and standard devia-
tions for all measures. No signi®cant differences in
pre-test questionnaire information were noted
between treatment and control groups, thus reducing
the selection internal validity concern. Correlation
analysis indicated a high degree of correlation
between pre-test Excel expertise and overall spread-
sheet expertise. Since the Excel expertise measure had

Table 4
Means and standard deviations (Study #2)

Variable Overall Ad hoc design (n 30) Struct. design n 58

Mean SD Mean SD Mean SD

Gender (1 Male, 2 Female) 1.38 0.49 1.37 0.49 1.38 0.49

Age group (1 20±25, 2 26±30, 3 31±40, 4 40±50) 1.18 0.54 1.19 0.56 1.19 0.54
Class level (1 Jr., 2 Sr., 3 Grad) 1.33 0.54 1.33 0.62 1.31 0.50
Domain expertise (1 Other, 2 Accounting) 1.93 0.25 1.93 0.27 1.90 0.31
Treatment group (1 Control, 2 Treat.) 1.66 0.48 1 0 2 0
Pre-test SS expertisea 2.86 0.75 2.91 0.48 2.79 0.83
Pre-test Excel expertisea 3.02 0.89 3.00 0.68 2.99 0.98
Pre-test DFD expertisea 2.35 0.79 2.41 0.69 2.36 0.83
Pre-test design confidenceb 3.11 0.86 3.04 0.75 3.14 0.93
Post-test SS expertisea 3.56 0.78 3.67 0.62 3.53 0.86
Post-test Excel expertisea 3.74 0.72 3.89 0.51 3.68 0.81
Post-test design confidenceb 3.25 1.09 3.11 0.93 3.36 1.15
Post-test coding confidenceb 3.08 1.13 2.69 0.93 3.28 1.18
Post-test error-free confidenceb 3.29 1.09 3.44 1.09 3.26 1.09
Mean number of linking errors/workbook 7.63 9.52 11.56 11.76 5.78 7.74
Linking error% 11.05% 0.14 16.75% 0.17 8.37% 0.11
Mean number of conceptual errors/workbook 0.38 0.59 0.37 0.56 0.38 0.62
Mean number of oversight errors/workbook 1.10 0.84 1.15 0.66 1.10 0.91
Mean number of total errors/workbook 9.07 9.66 12.96 11.97 7.26 7.87
Time (h) 19.00 6.62 18.13 7.35 19.38 6.36
Post-test DFD confidenceb 2.98 0.91
Post-test `DFD helped'c 2.84 0.99
Post-test `DFD took longer'c 2.83 0.99
Post-test `Will use DFD again'c 2.88 1.04
a
1 Novice, 5 Expert.
b
1 Very unconfident, 5 Very confident.
c
1 Strongly disagree; 5 Strongly agree.
 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12
Table 5
Regression analysis results (Study #2) N 88

Independent variables Standardized regression coefficients

Post SS Post. Post. Post. Post # Missing # Conc. # Oversignt # Total Total
expertise excel des. coding error links errors errors errors Time
expertise conf. conf. free conf.

Gender (1 M, 2 F) ÿ0.33a ÿ0.21 ÿ0.30 ÿ0.17 ÿ0.53a 1.04 0.14 0.42a 1.32 ÿ0.87
Age group 0.21 0.21 0.42 0.32 0.04 0.01 0.36 ÿ0.07 0.15 0.82
Class level (1 Jr., 2 Sr., 3 Grad.) ÿ0.13 ÿ0.07 ÿ0.34 ÿ0.31 ÿ0.49a 1.60 ÿ0.18 0.24 1.89 ÿ2.60
Domain experience (1 -Other, 2 Acctg.) 0.57a 0.38 1.02b 0.24 0.68 2.72 0.17 0.08 2.52 ÿ3.66
Treatment group (1 Control, 2 Treat) ÿ0.12 ÿ0.19 0.29 0.58a ÿ0.15 ÿ5.75b 0.06 0.00 ÿ5.63b 1.03
Pre. Excel expertise 0.30b 0.29b 0.30a 0.43b 0.06 ÿ3.18 ÿ0.07b ÿ0.15 ÿ3.22a ÿ1.11
Pre. DFD expertise 0.14 0.16 0.27a 0.16 0.04 ÿ0.34 0.47 0.02 ÿ0.23 ÿ0.64
Pre. design confidence 0.08 0.06 0.00 ÿ0.09 ÿ0.02 1.35 0.07 0.03 1.41 ÿ0.28
R2 0.32c 0.27b 0.31c 0.22a 0.20a 0.17a 0.21a 0.11 0.17a 0.09
a
p  0.05.
b
p  0.1.
c
p  0.001.

9
10 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12

Table 6
Regression analysis results for treatment attitudinal measures (Study #2) N 58

Independent variables Standardized regression coefficients

Post-test-treatment group only

DFD conf. DFD helpedd DFD took longerd Will use againd

Gender (1 M, 2 F) ÿ0.58b ÿ0.35 ÿ0.06 ÿ0.41

Age group ÿ0.04 ÿ0.37 ÿ0.45 0.18
Class levele ÿ0.16 0.45 ÿ0.09 0.17
Domain experiencef 0.64 ÿ0.24 ÿ0.33 ÿ0.61
Pre. SS/Excel expertise ÿ0.19 ÿ0.11 ÿ0.46b ÿ0.13
Pre. DFD expertise 0.41b 0.00 ÿ0.02 0.05
Pre. design confidence 0.26 0.41a 0.26 0.35a
R2 0.41c 0.17 0.15 0.19
a
p< 0.05.
b
p< 0.01.
c
p< 0.001.
d
1 Strongly disagree; 5 Strongly agree.
e
1 Jr., 2 Sr., 3 Grad.
f
1 Other, 2 Acctg.

the highest correlation with the dependent variables, it
was included in the regression analysis, and the pre-
test spreadsheet expertise measure was omitted. Table
5 shows the initial regression analysis for all variables.
Signi®cant results were attained for gender, pre-test
Excel expertise, DFD expertise, and design con®-
dence. A reduced model analysis performed by omit-
ting the non-signi®cant independent variables yielded
essentially the same results. Table 6 shows the regres-
sion analysis results for the design methodology atti-
tudinal measures obtained from the treatment group
subjects.
4.6. Summary and discussion
Apparently our independent variables accounted for
signi®cant differences in all of the dependent mea-
sures except number of conceptual errors and overall
time spent on design and coding. An interesting
gender difference surfaced: females reported signi®-
cantly (p  0.05) less perceived post-test spreadsheet
expertise and con®dence that their spreadsheets were
error-free. This concurs with Harrison and Rainer's
®nding that males tend to have more experience with
computers and higher con®dence levels. Interestingly,
the females also had signi®cantly (p  0.05) fewer
oversight errors. We postulated that this may have
been a result of their lower con®dence levels: perhaps
the females reviewed their spreadsheets more and
caught their oversight errors.
Subjects in higher class levels (i.e., seniors and
graduate students) also displayed less con®dence that
their spreadsheets were error free. There were no
notable correlations between class level and any of
the other independent variables; in fact, subjects in the
higher levels seemed, in general, to have less pre-test
application expertise. We attributed this result to the
inexperience and overcon®dence of youth.
Domain experience was a positive predictor of post-
test spreadsheet expertise (p  0.05) and post-test
design con®dence (p  0.01). Neither of these ®nd-
ings were particularly surprising. Domain experience
was somewhat correlated to pre-test spreadsheet
expertise (r 0.19) as well, so apparently the accounting
majors had more prior expertise with spreadsheets than
the non-accounting majors. The problem was fairly
complex, with ten different worksheets and a total of
66 links, so it is reasonable to expect that domain
experience would contribute to design con®dence.
Using the structured design methodology, the num-
ber of link errors reduced signi®cantly (p  0.01), but
had no impact on the number of oversight or con-
ceptual errors. Interestingly, link errors were by far the
most common. Each spreadsheet (for the combined
control and treatment groups) had an average of 9.07
total errors; of these, 84% were linking errors. It is
 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12
important to note that using the design methodology
had no signi®cant impact on total time spent on the
project (aside from the time taken to learn the meth-
odology).
Pre-test Excel expertise was the most signi®cant
predictor, positively impacting post-test spreadsheet
expertise (p  .01), post-test Excel expertise (p <
0.01), post-test design con®dence (p  0.05), and
post-test coding con®dence (p  0.01). Subjects with
higher pre-test Excel expertise also made fewer con-
ceptual errors (p  0.01) and fewer overall errors
(p < 0.01). These ®ndings concur with the results of
many studies emphasizing the importance of applica-
tion experience and resulting expertise e.g., [1,27].
The attitudinal responses indicate that females were
signi®cantly less con®dent of their post-test DFD
development ability. Pre-test design con®dence was
positively correlated to the belief that creating the
DFD was helpful and would be used in the future.
Subjects with a higher degree of pre-test spreadsheet
expertise did not think that developing the DFD
caused the development effort to take longer. Overall,
from an attitudinal standpoint, the subjects were
ambivalent of the costs and bene®ts of the structured
design approach.
5. Conclusions, limitations, and future research
Our experimental studies con®rm that developer
attributes, speci®cally gender and domain and appli-
cation expertise, signi®cantly impacted error rates.
Additionally, we show that using the structured design
methodology signi®cantly improved reliability for
two different linked spreadsheet applications. Study
#1 also inadvertently suggests that developer con®g-
uration is an important factor and has potential inter-
actions with the development approach.
Our initial study was confounded, because the pairs
were self-selected. Our second study removed this
possible bias. Additionally, our research uses absolute
error rates to proxy for the theoretical construct of
reliability. It is not clear how error rates in spread-
sheets are correlated with poor decisions resulting
from spreadsheet models. Thus, a construct validity
concern exists. Also, the potential interaction of pro-
blem complexity on the structured development
approach usage needs to be explored further.
11
Our theoretical framework identi®es several out-
comes associated with the use of end-user developed
spreadsheets. This research concentrates on outcome
reliability. The positive impact of development
approach cannot be generalized to usability, maintain-
ability, and/or audibility outcomes without more
research. Finally, the structured design approach used
explicitly identi®ed links during the design stage.
Other methodologies that explicitly specify potential
error factors prior to implementation may also
improve end-user application reliability.
As end-user development increases in organiza-
tions, the identi®cation of strategies to minimize risks
and improve outcomes is becoming increasingly
important. This research has made a contribution both
by identifying end-user development risk factors and
outcomes, and by developing and empirically evalu-
ating the impacts of a structured design approach for
spreadsheet development.
References
[1] B. Adelson, E. Soloway, The role of domain experience in
software design, IEEE Transactions on Software Engineering
11(11) (1985), pp. 1351±1360.
[2] M. Alavi, I. Weiss, Managing the risks associated with end-
user computing, Journal of Management Information Systems
2(3) (1986), pp. 413±426.
[3] D. Batra, A framework for studying human error behavior in
conceptual database modeling, Information and Management
25(3) (1993), pp. 121±131.
[4] R.S. Billings, S.P. Wroten, Use of path analysis industrial/
organizational psychology: Criticisms and suggestions, Jour-
nal of Applied Psychology 63 (1978), pp. 677±688.
[5] P.S. Brown, J.D. Gould, An experimental study of people
creating spreadsheets, ACM Transactions on Office Informa-
tion Systems 5 (1987), pp. 258±272.
[6] E. Cale, Quality issues for end-user developed software,
Journal of Systems Management 45 (1994), pp. 36±39.
[7] P. Cheney, R. Mann, D. Amoroso, Organizational factors
affecting the success of end-user computing, Journal of
Management Information Systems 3(1) (1986), pp. 65±80.
[8] L. Constantine, E. Yourdon, Structured Design, Prentice-Hall,
Englewood Cliffs, NJ, 1979.
[9] W. Cotterman, K. Kumar, User cube: a taxonomy of end
users, Communications of the ACM 32(11) (1989), pp. 1313±
1320.
[10] B. Curtis, H. Krasner, N. Iscoe, A field study of the software
design process for large systems, Communications of the
ACM 31(11) (1988), pp. 1268±1287.
[11] N. Davies, C. Ikin, Auditing spreadsheets, Australian
Accountant, 1987, pp. 54±56.
12 D. Janvrin, J. Morrison / Information & Management 37 (2000) 1±12

[12] F. Davis, Perceived usefulness, and user acceptance of [30] R.R. Panko, R.P. Halverson, Spreadsheets on trial: a survey of
information technology, MIS Quarterly 13(3) (1989), pp. research on spreadsheet risks, in Proceedings of the 29th
318±339. Hawaii International Conference on System Sciences, Vol. II,
[13] F. DeRemer, H. Kron, Programming-in-the-large versus IEEE Computer Society Press, 1996, pp. 326±335.
programming-in-the-small, IEEE Transactions on Software [31] R.R. Panko, R.P. Halverson, Are two heads better than one (at
Engineering 2(1) (1976), pp. 80±86. reducing spreadsheet errors in spreadsheet modeling)?, Office
[14] W. Doll, W. Torkzadeh, A discrepancy model of end-user Systems Research Journal 15 (1997), pp. 21±32.
computing involvement, Management Science 35(10) (1989), [32] R.R. Panko, R.H. Sprague, Hitting the wall: errors in
pp. 1151±1171. developing and code inspecting a `simple' spreadsheet model,
[15] J. Frank, B. Shamir, W. Briggs, Security-related behavior of Decision Support Systems 22 (1998), pp. 337±353.
PC users in organizations, Information and Management. [33] K. Rajalingham, D. Chadwick, Integrity control of spread-
21(4) (1991), pp. 127±135. sheets: Organization & tools, in: S. Jajodia, W. List, G.
[16] C. Gane, T. Sarson, Structured Systems Analysis, Prentice- McGregor, L. Strous (Eds.), Integrity and Internal Control in
Hall, Englewood Cliffs, NJ, 1979. Information Systems, Kluwer Academic Publishers, Boston,
[17] A. Harrison, R.K. Rainer, The influence of individual MA, 1998, pp. 147±168.
differences on skill in end-user computing, Journal of [34] J. Rockart, L. Flannery, The management of end user
Management Information Systems 9(1) (1992), pp. 93±111. computing, Communications of the ACM 26 (1983), pp.
[18] D.R. Heise, Problems in path analysis and causal inference, 776±784.
in: E.F. Borgatta (Ed.), Sociological Methodology, Jossey- [35] B. Ronen, M. Palley, H. Lucas, Spreadsheet analysis and
Bass, San Francisco, CA 1969, pp. 38±73. design, Communications of the ACM 32 (1989), pp. 84±93.
[19] J. Henderson, M. Treacy, Managing end-user computing for [36] D. Ross, K. Schoman, Structured analysis for requirements
competitive advantage, Sloan Management Review 28 definition, IEEE Transactions on Software Engineering 3(1)
(1986), pp. 3±14. (1977), pp. 6±15.
[20] D.G. Hendry, T.R.G. Green, Creating, Creating and compre- [37] L. Salchenberger, Structured development techniques for
hending and explaining spreadsheets: a cognitive interpreta- user-developed systems, Information and Management 24(1)
tion of what discretionary users think of the spreadsheet (1993), pp. 41±50.
model, International Journal of Human-Computer Studies 40 [38] E.J. Savitz, Magellan loses its compass, Barron's 84(12)
(1994), pp. 1033±1065. (1994), pp. 50.
[21] R. Hilton, Managerial Accounting, 2nd ed., McGraw Hill, [39] M.G. Simkin, How to validate spreadsheets, Journal of
New York, NY, 1994, pp. 439±440. Accountancy 164(5) (1987), pp. 130±138.
[22] M. Igbaria, S. Parasuraman, A path analytic study of
individual characteristics, computer anxiety and attitudes
toward microcomputers, Journal of Management 15(3)
(1989), pp. 373±387.
[23] I. Lorge, H. Solomon, Two models of group behavior in the
solution of Eureka-type problems, Psychometrika 20(2)
(1955) (cited in [29]).
[24] L. Markus, M. Keil, If we build it, they will come: designing
information systems that people want to use, Sloan Manage-
ment Review 36 (1994), pp. 11±25.
[25] S. Moriarity, C. Allen, Cost Accounting, 3rd ed., Wiley, New
York, NY, 1991, pp. 285±291. Diane Janvrin
[26] B. Nardi, J. Miller, Twinkling lights and nested loops: distributed
problem solving and spreadsheet development, International
Journal Man-Machine Studies 24 (1991), pp. 161±184.
[27] O. Ngwenyama, Developing end-users' systems development
competence, Information and Management 25(6) (1993), pp.
291±302.
[28] R.R. Panko, What we know about spreadsheet errors, Journal
of End User Computing 10 (1998), pp. 15±21.
[29] R.R. Panko, R.P. Halverson, Individual and group spreadsheet
design: Patterns of Errors, in Proceedings of the 27th Hawaii
International Conference on System Sciences, Vol. IV, IEEE
Computer Society Press, 1994, pp. 4±10. Joline Morrison