JEIM_cover_(i).

qxd 8/5/04 3:08 PM Page 1

Volume 17 Number 4 2004 ISBN 0-86176-996-1 ISSN 1741-0398

Journal of

Enterprise
Information
Management
Formerly Logistics Information Management:
An International Journal

Information systems evaluation: getting closer to the

organisation
Guest Editors: Dr Stephen Jones and Dr Jim Hughes

www.emeraldinsight.com
 Journal of Enterprise Information Management
Volume 17, Number 4, 2004 ISSN 1741-0398

Information systems evaluation: getting closer

to the organisation
Guest Editors: Dr Stephen Jones and Dr Jim Hughes

Contents
242 Access this journal online
243 Abstracts & keywords
245 Guest editorial
249 An integrative, iterative approach
to IS evaluation for
pharmaceutical clinical trials
Catherine Dymoke-Bradshaw and
Benita Cox
258 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
269 A hedonic wage model-based
methodology for evaluating the
benefits of IT investments
in public-sector organisations
Peter Cilek, Wolfgang Janko,
Stefan Koch, Andreas Mild and
Alfred Taudes
276 Information systems evaluation
and the information systems
development process
Paul Beynon-Davies, Ian Owens and
Michael D. Williams
283 Narratives in ERP systems
evaluation
Jonas Hedman and Andreas Borell
Other papers
291 Online point-of-click Web
usability mining with PopEval
(C-Assure Co-design
methodology)
Atta Badii
301 Benefits and barriers of electronic
marketplace participation: an
SME perspective
Rosemary Stockdale and Craig Standing
312 Information technology
evaluation: classifying indirect
costs using the structured case
method
Peter E.D. Love, Ahmad Ghoneim and
Zahir Irani

Access this journal electronically

The current and past volumes of this journal are available at:
www.emeraldinsight.com/1741-0398.htm
You can also search over 100 additional Emerald journals in Emerald Fulltext:
www.emeraldinsight.com/ft
See page following contents for full details of what your access includes.
www.emeraldinsight.com/jeim.htm
As a subscriber to this journal, you can benefit from instant,
electronic access to this title via Emerald Fulltext. Your access
includes a variety of features that increase the value of your
journal subscription.
How to access this journal electronically
To benefit from electronic access to this journal you first need
to register via the Internet. Registration is simple and full
instructions are available online at www.emeraldinsight.com/
rpsv/librariantoolkit/emeraldadmin Once registration
is completed, your institution will have instant access
to all articles through the journal’s Table of Contents
page at www.emeraldinsight.com/1741-0398.htm More
information about the journal is also available at
www.emeraldinsight.com/jeim.htm
Our liberal institution-wide licence allows everyone within
your institution to access your journal electronically, making
your subscription more cost effective. Our Web site has been
designed to provide you with a comprehensive, simple system
that needs only minimum administration. Access is available
via IP authentication or username and password.
Key features of Emerald electronic journals
Automatic permission to make up to 25 copies of
individual articles
This facility can be used for training purposes, course
notes, seminars etc. This only applies to articles of which
Emerald owns copyright. For further details visit
www.emeraldinsight.com/copyright
Online publishing and archiving
As well as current volumes of the journal, you can also gain
access to past volumes on the internet via Emerald Fulltext.
Archives go back to 1994 and abstracts back to 1989. You can
browse or search the database for relevant articles.
Key readings
This feature provides abstracts of related articles chosen by
the journal editor, selected to provide readers with current
awareness of interesting articles from other publications in
the field.
Reference linking
Direct links from the journal article references to abstracts of
the most influential articles cited. Where possible, this link is
to the full text of the article.
E-mail an article
Allows users to e-mail links to relevant and interesting articles
to another computer for later use, reference or printing
purposes.
Additional complementary services available
Your access includes a variety of features that add to the
functionality and value of your journal subscription:
E-mail alert services
These services allow you to be kept up to date with the latest
additions to the journal via e-mail, as soon as new material
enters the database. Further information about the services
available can be found at www.emeraldinsight.com/
usertoolkit/emailalerts
Emerald WIRE (World Independent Reviews)
A fully searchable subject specific database, brought to you
by Emerald Management Reviews, providing article reviews
from the world’s top management journals.
Research register
A Web-based research forum that provides insider
information on research activity world-wide located at
www.emeraldinsight.com/researchregister You can also
register your research activity here.
User services
Comprehensive librarian and user toolkits have been created
to help you get the most from your journal subscription.
For further information about what is available visit
www.emeraldinsight.com/usagetoolkit
Choice of access
Electronic access to this journal is available via a number of
channels. Our Web site www.emeraldinsight.com is the
recommended means of electronic access, as it provides fully
searchable and value added access to the complete content
of the journal. However, you can also access and search the
article content of this journal through the following journal
delivery services:
EBSCOHost Electronic Journals Service
ejournals.ebsco.com
Huber E-Journals
e-journals.hanshuber.com/english/index.htm
Informatics J-Gate
www.j-gate.informindia.co.in
Ingenta
www.ingenta.com
Minerva Electronic Online Services
www.minerva.at
OCLC FirstSearch
www.oclc.org/firstsearch
SilverLinker
www.ovid.com
SwetsWise
www.swetswise.com
TDnet
www.tdnet.com
Emerald Customer Support
For customer support and technical help contact:
E-mail support@emeraldinsight.com
Web www.emeraldinsight.com/customercharter
Tel +44 (0) 1274 785278
Fax +44 (0) 1274 785204

Abstracts & keywords
An integrative, iterative approach to IS evaluation
for pharmaceutical clinical trials
Catherine Dymoke-Bradshaw and Benita Cox
Keywords Information systems,
Pharmaceuticals industry
This paper proposes an “innovative” approach to the
evaluation of information systems projects to support
pharmaceutical clinical trials. The IS evaluation
process for these projects closely follows that
described in the Management of Innovation literature
using methods which are highly iterative and are an
integral component of the project. The development
of organization-technology fit is core and evaluation
for this purpose can be thought of as a coupling
mechanism between use and design and as a tool for
continuous learning and improvement. This
integrative, iterative approach is applied to a project
undertaken at Glaxo Wellcome Research and
Development.
Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
Keywords Cost benefit analysis,
Investment appraisal, Communication technologies
The paper presents an approach for defining ICT
investment proposals. The approach was developed
on the authority of the Dutch Ministry of Transport,
Public Works and Water Management. The Ministry
was confronted with a growing amount of ICT
investment proposals. Since there was no format for
describing a proposal and no procedure for obtaining
the data on the basis of which a proposal was
described, the decision to select proposals for funding
missed any kind of rationality. Proposals differed
enormously and hence were incomparable.
Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · Abstracts & keywords
q Emerald Group Publishing Limited · ISSN 1741-0398
243
The evaluation of existing ICT assessment methods
showed that there was not any method that would
meet the expectations and demands of the Ministry.
The main points of criticism concentrated on the
limitation of applying a fixed set of data for describing
a project proposal and the absence of a process to
support the selection of a meaningful set of data to
describe a project. Inspired by the multi-criteria score
method of Parker a tailor-made approach for defining
ICT investment proposals was developed, taking into
account the characteristics of the governmental
organization. In this paper, the approach as well as the
developmental process with regard to this approach
are presented. The overall conclusion is that a tailor-
made approach, which is developed with the full
cooperation of the user organization, has considerable
advantages since local knowledge and expertise is
incorporated in this type of approach.
A hedonic wage model-based methodology
for evaluating the benefits of IT investments
in public-sector organisations
Peter Cilek, Wolfgang Janko, Stefan Koch, Andreas Mild
and Alfred Taudes
Keywords Information systems,
Public sector organizations, Investment appraisal,
Cost benefit analysis
The economic justification of investments in
information technology (IT) is a basic issue for IT
management in private and in public-sector
organisations. Usually, the expenses made for any
investment are compared to the cost saved. While the
costs for implementing a new system are uncertain,
only a small percentage of the benefits accrues as cost
savings given the type of IT systems used today and
the particularities of the public-sector. In this paper,
we present a methodology for the monetary
quantification of the benefits resulting from the
introduction of a modern IT application and
demonstrate its use on the basis of a case of prison
administration.
Information systems evaluation and the
information systems development process
Paul Beynon-Davies, Ian Owens and Michael D. Williams
Keywords Information systems, Modelling,
Life cycle costs
In this paper, we consider the synergy between two
areas of information system (IS) literature: that
concerned with the evaluation of IS and that
concerned with explaining the phenomenon of IS
failure. On the basis of an analysis of both areas,
a model is presented which attempts to integrate IS
evaluation into the life cycle of IS development.
The model links the issue of failure assessment with
the evaluation process and constitutes a strategy for
 Abstracts & keywords
stimulating organisational learning in relation to IS
development. The paper concludes with a description
of our attempts to validate aspects of the model and
plans for further empirical work in this area.
Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
Keywords Narratives, Information systems,
Manufacturing resource planning
The purpose of evaluating enterprise resource
planning (ERP) systems ought to be to improve the
systems’ life cycles. In this paper, narratives are
proposed as a means of improving ERP systems as a
complement to traditional evaluation methods.
The potential of narratives is that they can convey
meanings, interpretations, and knowledge of the
system, which may lead to action. Even though
narratives belong to an interpretive research tradition,
this paper takes a pragmatic view of evaluation on the
basis of three assumptions about evaluation:
evaluations should form the basis for action;
narratives can make evaluation more relevant; and
evaluations should be made with the purpose of
improving selection, implementation and use of the
system. The conclusion reached is that narratives can
advance evaluation practice by providing a richer
evaluation picture which conveys meanings not
included in traditional evaluations, and improve the
use of ERP systems by changing users’ mental maps.
Online point-of-click Web usability mining with
PopEval (C-Assure Co-design methodology)
Atta Badii
Keywords Online operation, User interfaces,
User satisfaction
Earlier research has shown that human judgements,
particularly retrospective as distinct from real-time
evaluations of painful experiences, are fundamentally
flawed when superseded by other experiences.
Describes a methodology for the evaluation of user
experiences, assisted by a user-centered co-design tool
for planning, creating and conducting wide-ranging
usability data acquisition throughout the system life
cycle from inception to replacement. The results from
the tests in this study were found to support the
findings from recent cognitive psychology research on
J/DM-PPR theoretic biasing effects. The importance
of the findings for software systems, and for interactive
Web site development, is that if the interactive features
which tax the user’s patience and cognitive capability
are not severely restraining, then users will be left with
a better impression of the usability of the system than
would be the case otherwise.
244
Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 243 – 244
Benefits and barriers of electronic marketplace
participation: an SME perspective
Rosemary Stockdale and Craig Standing
Keywords Small to medium-sized enterprises,
Electronic commerce, Trade barriers
There are concerns that despite government
initiatives to promote adoption of electronic
commerce, SMEs still fail to realise e-commerce
related benefits. It may therefore, seem premature to
discuss electronic marketplaces in the context of
SMEs. However, if SMEs ignore e-marketplaces a
number of problems can result. E-marketplaces
present a significant threat to SMEs since they
increase competition and leave non-participants
vulnerable to more e-enabled firms. This paper
examines the barriers and benefits of e-marketplace
participation by SMEs. The nature of e-marketplaces
is addressed and the benefits of participation are
examined. Drawing on the literature, the barriers
facing smaller firms in this environment are discussed.
Identification of these barriers, such as lack of
standards, supply chain integration and global
trading, enables a greater understanding of how
SMEs can plan effective strategies to gain from
e-marketplace participation.
Information technology evaluation: classifying
indirect costs using the structured case method
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
Keywords Electronic commerce,
Competitive advantage, Costs, Research methods
Many companies are increasing their expenditure on
information technology and information systems
(IT/IS) to obtain or even sustain a competitive
advantage in their respective marketplaces. Many
managers, however, are often left with the quandary
of how to evaluate their investments in technologies.
Reasons of this difficulty have been suggested in the
normative literature as centring on the socio-technical
(human, organisational and technical) dimensions
associated with the adoption of IT/IS. The inability of
managers to determine the true costs of deploying
IT/IS is considered attributable to a lack of knowledge
and understanding of IT/IS related costs. In
developing a broader picture of such cost dimensions
and their respective taxonomies, the research
presented in this paper uses a structured case method
to gain an understanding of how a construction firm
embraced the IT evaluation process. A review of the
IT cost literature is presented and a conceptual IT
evaluation framework that focuses on indirect costs is
proposed.

Guest editorial
About the Guest Editors
Stephen Jones is the Chief Information Systems Officer and
heads the Information Technology Department of Conwy County
Borough Council, a Welsh Unitary Authority in the UK. He has
worked in several roles in a career over 25 years as an IS
practitioner, including programming, systems analysis, project
management, systems development management, consultancy
and IT Departmental Head. His work has been predominately in
the public sector, where he has worked for several large
organisations. He holds a BSc with first class honours in Business
Information Systems Management from Salford University, a
MSc with distinction in Information Systems from Liverpool
University and a PhD in Information Systems from Salford
University. He is a chartered engineer (CEng) and a member of
the British Computer Society (MBCS), Chartered Institute of
Management (MCMI) and Society of Information Technology
Managers (SOCITM). He was instrumental in establishing an IT
benchmarking initiative for IT Departments in Welsh Unitary
Authorities and undertook preparatory work to define Key
Performance Indicators (KPI’s) for the IT service in Local
Government. Dr Jones has undertaken guest lectures at several
UK Universities in the area of IS management, IS evaluation,
systems analysis, project management, e-Government, IS ethics
and IS law. He is a member of several academic and practitioner
working groups. He sits on editorial boards, has edited
international journals and written book chapters. He has written
several refereed papers at national and international
conferences, has chaired conference sessions and serves on
international conference committees. His research interest is in
the area of IT investment appraisal and evaluation.
Jim Hughes works in the UK for the Cheshire and Merseyside
Strategic Health Authority as Head of Knowledge Management.
He has a degree in Computer Science from Liverpool University
and a PhD in Information Systems from the University of Salford.
He has worked as a software engineer and systems analyst for
the Ford Motor Company and has consulted extensively in the
public and private sectors to a number of small and medium
sized enterprises in the fields of information systems and
knowledge management. Jim was until recently a lecturer and
researcher at Salford University in the field of Information
Systems, and retains his research links there. He has published
over 30 papers. Whilst at Salford he founded a spin-out
knowledge management technology company. Jim is particularly
interested in the situated – contextual – nature of information
and in the course of his PhD, he explored a methodology for
eliciting situated requirements. Jim’s research work is
interpretive and action-based. An interest in the situated nature
of information has led to his involvement in considering a
situated hermeneutic approach to assessing IS value in the IS
evaluation process.
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 245–248
q Emerald Group Publishing Limited · ISSN 1741-0398
245
Information systems evaluation: getting
closer to the organisation
The deployment of information systems (IS) has
been heralded as the solution to a myriad of
problems associated with the modern business and
organisation. IS industry observers predict that the
significant global IS expenditure will continue to
increase. However, despite this investment, there is
widespread concern, especially amongst senior
executives, that IS investment does not always
deliver value or realise the anticipated benefits.
Indeed, it is often perceived that IS
implementation leads to disbenefits. This
phenomenon has been labelled the “IT
productivity paradox” and is most accurately
linked to IS research studies in the early nineties.
IS expenditure, is usually both costly and risky,
and any IS investment should therefore, be
examined for its contribution, value and benefit to
an organisation. How then, can IS be evaluated.
The Oxford English Dictionary defines
evaluation as “the action of working out the value
of something”. This leads to the notion that
evaluation is a subjective process, which is
undertaken to assess the value of an object or the
merit of a situation. That is to say that
measurement is not evaluation. This is an
important issue, which is often overlooked in IS
evaluation processes.
Traditionally, evaluation has been associated
with management practice, in an attempt to
improve management decision-making and
accountability, and organizational efficiency and
effectiveness. Both intuitive and explicate
evaluation is an integral part of the role of
management. This evaluation helps to allocate
resources correctly to maintain or improve
performance. For example, large capital
expenditures are rarely made without an
accompanying economic evaluation. However, the
literature illustrates that IS evaluation has not been
undertaken to the same extent.
Organisations often tacitly assume that IS
evaluation is the task of the IS function. However,
in practice IS evaluation has been given little
attention, with the emphasis being on project
management, systems analysis and programming.
Moreover, where IS evaluation is undertaken by
the IS function, the focus is usually upon the
technical aspects. However, even though it may be
possible to demonstrate that IS are performing
well from a technical perspective, this does not
necessarily lead to successful use or organizational
improvements.
It is often perceived that it is difficult to
effectively evaluate IS unless there is a clearly
documented, systematic and formal approach.
 Guest editorial
However, many organisations have no
management processes to govern and measure the
outcome or processes to establish what benefits
were actually achieved. Where IS evaluation has
been undertaken, traditional, formal, quantitative,
mechanistic methods based on economics have
been employed.
At first sight this approach appears to be both
necessary and appropriate and it would appear
difficult to argue against this method conceptually.
However, in practice this is problematic because it
is difficult to predict IS costs, risks, benefits and
impact. Managers have difficulty in determining
what aspect to measure, with the possible
exception of economic metrics. However, although
appearing to be relatively straightforward to
calculate, these financial factors often become
problematic during the subsequent analysis. This
situation makes it difficult to assess the impact and
contribution of IS. Furthermore, what is of
importance is not whether IS is cost justified, but
rather extent to which IS is useful and successful in
practice.
This is particularly true of the UK public sector,
where IS evaluation has traditionally been
neglected and where economic value has little
meaning. However, the UK central government
has recently introduced the Best Value and
Corporate Improvement initiative, which is rapidly
raising the profile and importance of evaluation in
general, including IS evaluation. IS evaluation
theory and practice is now a key area, especially in
the public sector.
Various models have been proposed to measure
IS efficiency and effectiveness. However, there
seems to be little consensus among researchers and
practitioners as to which approach is appropriate.
Many IS practitioners and IS stakeholders are not
concerned with the specifics of IS assessment
metrics, detailed benefit measurement formulae or
mechanistic evaluation techniques. They are
concerned with the successful outcomes, in terms
of the introduction, operation and impact of IS.
These issues should be considered to enable an
organisation to gauge the value of any IS
implementation.
We maintain that evaluation is a social
process, as individuals have important opinions
based upon their own worldview. Evaluation
therefore, occurs in everyday life and individuals
evaluate based on their knowledge, experience,
background, understanding and intuition. These
are not formal, overt, evaluation processes but
informal, covert processes. Nonetheless, they have
significant importance to the individuals and peer
groups concerned. With regard to IS evaluation,
these opinions may be of importance to an
organisation.
246
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 245–248
Our interests are in the area of IS evaluation,
particularly in the public sector, and in the
multidisciplinary and social aspects of IS and these
interests were expressed in the call for papers. In
the process of selecting the papers for inclusion in
the special issue we were conscious of the need to
include a range of papers in which we encouraged
articles that fell outside of the dominant paradigm.
We feel this has been achieved. The five papers
each suggest either innovative approaches to the
process of Information Systems evaluation or
present new ways of looking at old problems.
Through the papers as a whole two key themes
emerge which we believe are significant enough to
be taken seriously by those with an interest in this
area. First, the theme of “narrative” or “story”.
This is popularised in the interpretive tradition as a
means of presenting organizational richness in a
form that enables the reader to interpret and learn.
Within this theme the case study is perhaps the
most common method of “telling the
organizational story” and whilst the author may
provide analysis and discussion the reader also
engages by making their own connections from
their own experience and background. The second
theme is one that we term “situated”. In this theme
a number of the papers are concerned with
methods that cannot be seen as prescriptions.
Here we are concerned with the context in
which methods are used and are open to
the idea that established methods need to
be critically challenged and then adapted for
specific local situations. The driver for the
method is not its popularity, but its
appropriateness to the situation and the learning
that emerges. For each of the papers, we next
provide a synopsis and consider their contribution
to these themes and to the field of Information
Systems evaluation.
Dymoke-Bradshaw and Cox propose an
approach to Information Systems evaluation
that has been developed specifically to deal with
Information Systems projects that support
pharmaceutical trials. Their research identifies
that the evaluation processes that are associated
with traditional systems development life cycle are
lacking because of their emphasis on externality
and also neutrality. In the specific context of the
pharmaceutical trials the evaluation process itself
is a highly integrated and dynamic part of the
project. Specifically their observations in this
domain led them to realize that the technology
products underlying the projects were subject to
change, whereas traditionally they are considered
to be stable. They develop a framework for the
pharmaceutical trials projects in which they
propose an iterative approach to the frequency and
timing of evaluation exercises. In their illustrative
 Guest editorial
case study with Glaxo Wellcome Research and
Development they describe an electronic data
capture project to implement electronic data
capture tools in clinical trials. The key learning
point that emerges from this case is that, it is only
through the practical actions associated with the
evolution of the project that a deep understanding
of requirements emerges. Additionally underlying
technology may also be changing and these
changes must also be accommodated. There are
important lessons here related to the themes
above. First, the case study story engages the
reader to question his or her own practice.
Second, we are presented with a situation in
which our intuitive assumptions regarding
evaluation, requirements and technology are
challenged. Dymoke-Bradshaw and Cox
suggest that their research may be useful to
practitioners and academics if it helps them to
question their assumptions about IS evaluation
and be brave enough to adapt traditional methods
when the organizational circumstances suggest
they should.
Heemstra and Kusters’ paper is concerned with
an approach for defining investment proposals for
Information Communication and Technology
(ICT) projects. The organisation they worked
with, The Dutch Ministry of Transport, Public
Works and Water Management, had no means by
which they could assess a growing number of
disparate ICT investment proposals that they
received. The authors’ research led them to the
view that despite the number of methods
explicated in the literature there was not a single
method that would satisfactorily meet the
demands of the Ministry. In the face of these
challenges the authors took a candidate method
(Parker’s multi-criteria score method) as a starting
point and tailored it for the specific organisation.
Their paper takes the reader through the process
of adoption. They note that a key to producing the
appropriate method was the full cooperation of the
organizational stakeholders since it is they who
have the specific local knowledge that an outsider
could not possibly have. They outline two key
lessons from their research. The first lesson relates
to data and is concerned with being able to answer
for the specific domain what data needs to be used
for the evaluation. They lament the fact that many
traditional approaches answer this simply by
suggesting financial data. The second lesson
relates to the process of evaluation. During the
process they took the perspective that the process
of evaluation was a human information-processing
activity and not simply a mechanistic or formal
process. In terms of the themes for this special
issue the main resonance is that associated with
situation and attention to context. Heemstra and
247
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 245–248
Kusters urge the readers to understand each
situation, engage the organizational stakeholders
and adapt the methods where needed.
Cilek et al., in their paper present a
methodology based on the hedonic wage model to
provide a monetary quantification of the benefits
that result from the introduction of an IT
application. In developing the methodology, the
authors consider the particular circumstances of a
public-sector organisation and use as their
illustrative case study the introduction of an IT
system covering several key administration systems
in a prison. The authors argue that in such
environments other methods of monetary
quantification would not be possible because they
depend on productivity gains being measured
against employee lay-offs. In their methodology,
Cilek et al., calculate hedonic prices from external
effects using historic data. What is interesting
about the paper is that the authors have identified
that a model extensively used in the private
sector becomes unworkable in the public sector
because of the particular set of circumstances
that prevail in such situations. They tentatively
suggest that their methodology could be of use
in other public sector organisations or used for
evaluating process change in organisations.
Benyon-Davis Owens and Williams’ paper is
more theoretical in nature. They consider two
areas of academic literature – IS evaluation and
the phenomena of IS failure. They propose a
model which links assessment of IS failure with the
IS evaluation process. They go on to develop a
strategy that integrates IS evaluation into the
systems development life cycle. Through the
literature they identify that many organisations do
not learn from failure other than by modifying
strategies and assumptions within accepted
normative boundaries. An important part of their
proposed model is that evaluation processes are
closely linked to IS development activities to
encourage organizational learning. They go on to
distinguish between strategic formative,
summative and post-mortem evaluation as key
processes in facilitating the learning of
stakeholders. What is interesting about the paper is
that the authors explicitly draw attention to IS
evaluation as a situated process.
Hedman and Borell’s paper is also theoretical in
nature. They propose the use of narratives to
complement traditional evaluation methods in
Enterprise Resource Planning systems. They
contend that narratives as an intuitive means of
expression can help in IS evaluation by conveying
meaning and knowledge about a system. They
outline their assumptions about IS evaluation with
a key assumption claiming that narratives can give
a better understanding of ERP systems than
 Guest editorial
methods commonly associated with traditional IS
evaluation. They note that narratives can be
written at any time before during or after a project
and that the crucial role that they play is that one
can keep track of meta-information and
interdependencies in the project. The authors
identify different narrative (story) types and the
ways in which these may be used to support,
persuade, overcome problems and disseminate
knowledge. The paper is strongly embedded in the
interpretive paradigm and for readers it may be
challenging since it proposes a new, perhaps even
tangential, way of looking at familiar problems.
248
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 245–248
Hedman and Borell conclude their paper by
focusing on the knowledge sharing and learning
characteristics of the narrative approach that are
particularly appropriate for ERP, and other such
large complex IS implementations that have such
a profound affect on organizational life.
Dr Stephen Jones
Chief Information Systems Officer, Conwy County
Borough Council; E-mail: steve.jones@
conwy.gov.uk
Dr Jim Hughes
Head of Knowledge Management, Cheshire and
Merseyside Strategic Health Authority

An integrative, iterative
approach to IS
evaluation for
pharmaceutical clinical
trials
Catherine Dymoke-Bradshaw and
Benita Cox
The authors
Catherine Dymoke-Bradshaw is based at Glaxo SmithKline,
Middlesex, UK.
Benita Cox is based at The Management School, Imperial
College of Science, Technology and Medicine, London, UK.
Keywords
Information systems, Pharmaceuticals industry
Abstract
This paper proposes an “innovative” approach to the evaluation
of information systems projects to support pharmaceutical
clinical trials. The IS evaluation process for these projects closely
follows that described in the Management of Innovation
literature using methods which are highly iterative and are an
integral component of the project. The development of
organization-technology fit is core and evaluation for this
purpose can be thought of as a coupling mechanism between
use and design and as a tool for continuous learning and
improvement. This integrative, iterative approach is applied to a
project undertaken at Glaxo Wellcome Research and
Development.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 249–257
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548652
249
Introduction
This paper proposes an “innovative” approach to
the evaluation of information systems (IS) to
support pharmaceutical clinical trials. The need for
such an approach results from an observation that
the methods used for evaluation of the IS to support
these trials differ from those described in the
current academic literature in a number of ways.
First, the IS evaluation process traverses three
major phases – initiation, adoption and
implementation (Rogers, 1983). This process is
akin to that described in the Management of
Innovation literature and differs from evaluation
exercises conducted for those projects that follow a
traditional systems development life cycle (SDLC)
development process. Whilst similarities exist in the
evaluation methods used during the initiation and
adoption phases of the process, differences in
emphasis occur during the implementation stage.
Traditional evaluation methods tend to view
implementation as the final step in the process
which is hardly elaborated upon. Evaluation in the
literature implies that once a system is developed
and implemented it is “good” if it requires little
subsequent change. Our observations suggest that
in pharmaceutical clinical trials it often is not until
practical experience in using the system has been
gained that the requirements are really understood.
Evaluation methods described for traditional
SDLC projects are broadly described as “external”
to the project. In pharmaceutical clinical trials the
IS evaluation process is both tightly integrated
within the project and highly dynamic – a kind of
“bootstrapping” process of learning as you go and
there is a tight coupling between the design and use
of the IS product or service (Orlikowski, 1992).
Further, the current evaluation literature says little
about changes in the underlying technology
products, implicitly assuming them to be stable,
whereas we observed that not only were products
changing during the evaluation process but that
opportunities were sought to influence the
direction of such change.
There are many similarities in the purpose of the
evaluation undertaken for SDLC projects and IS
projects to support clinical trials, however, there are
also a number of differences. In SDLC projects,
evaluation may be carried out for the purposes of
justification, as part of project control or for
learning for future projects or evaluation exercises
(Farbey et al., 1993). However, in clinical trial
projects these purposes are extended to include
evaluation of organization-technology fit where the
evaluation process itself can be thought of as a
mechanism for achieving coupling between design
The authors would like to thank Glaxo Wellcome
R&D for their support in sponsoring this research.
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
and use and as a tool for continuous learning and
improvement. Evaluation for the purpose of
strategic justification of the project is important in
both types of project, however, equally important in
clinical trials is evaluation to support decision
making at an operational level. For example, to
support decisions on options within the project,
such as the funding of feasibility studies, the setting
up of pilots or the approach to be taken to initial
implementation and subsequent roll-out.
The timing and frequency of the evaluation
process also differs. The evaluation process is
similar in the very early stages of the project and in
the later, post-implementation phases of routine
operational use. However, significant differences
occur in that in SDLC projects, the development
and implementation phases are separate, with
evaluation occurring during development but
conducted less rigorously during implementation.
In clinical trial projects, however, development
and implementation become tightly integrated and
a high level of evaluation during implementation is
essential because it is the feedback mechanism for
continuous learning. The evaluation exercises are
iterative and there are likely to be a number of
separate decisions that will be subjected to review
as circumstances change (Figure 1).
In this paper, an approach to the evaluation of
IS projects to support pharmaceutical clinical trials
Figure 1 The purpose and timing of evaluation in PCT projects
250
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
is developed and applied to a case study of
electronic data capture carried out at Glaxo
Wellcome Research and Development.
Purpose and timing of IS evaluation
exercises
The purpose of IS evaluation is primarily to assess
the costs and benefits of IS investments (Remenyi
et al., 1993). Other reasons for evaluation cited in
the academic literature include justification for an
existing or proposed new system, comparison
between projects competing for resources or to
provide benchmarks and control procedures
(Baker, 1995; Clemons, 1991; Farbey, et al., 1995;
Hawgood and Land, 1988; Targett, 1996;
Willcocks and Graeser, 2001). Some authors cite
the use of the evaluation process as a tool for
organisational learning (Argyris and Schon, 1996;
Earl, 1989; Galliers, 1991) or for understanding
and learning about an IS which is already in use
(Edzerodt and Madsen, 1988) or for political
reasons (Gregory and Jackson, 1992; Powell, 1992;
Symons, 1993; Walsham, 1993). Serafeimidis and
Smithson (2000) suggest that a further purpose of
IS evaluation might be to learn about the
organisation and its business dimensions.
Whilst the purposes of the evaluation exercises in
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
clinical trials are, in many instances, the same as
those highlighted in the academic literature, we
noted a number of further purposes, namely to
assess the extent of organisation-technology fit, for
project justification and decision making at an
operational as well as a strategic level and as an
iterative tool for project monitoring and control.
Similarities and differences also occur in the
timing and frequency of evaluation. It is generally
recognised that evaluation should be carried out at
all stages of an IS project and many authors
distinguish between appraisals which are
conducted ex ante, ex post or continuously
throughout the project life cycle (Earl, 1989;
Farbey et al., 1992). However, organisations tend
to complete evaluation at the feasibility stage, but
there is then a fall-off in later stages (Remenyi et al.,
1993; Willcocks, 1994). A number of studies
suggest that there are linkages between the timing
and purpose of evaluation exercises (Hochstrasser
and Griffiths, 1991; Kumar, 1990). In clinical
trials project, however, the evaluation process is
conducted iteratively throughout all stages of the
project life cycle. This process is similar to that of
the formative evaluation methods described for
active benefits realisation which encapsulates the
notion of continuous assessment and co-evolution
(Remenyi et al., 1997).
Purpose and timing of IS evaluation for
clinical trials
In the following section, a framework which
integrates the purpose and timing of evaluation
exercises is proposed. It extends the existing
literature by incorporating additional purposes for
evaluation and proposes an iterative approach to
the frequency and timing of evaluation exercises.
The framework was developed specifically for
pharmaceutical clinical trials, but we believe it also
has applicability for similar projects which do not
follow a traditional systems development life cycle
in other sectors.
Organization-Technology Fit
An important requirement of IS projects for
pharmaceutical clinical trials is to achieve a close
fit between the organization and the technology
(Orlikowski, 1992). People use technology and
therefore, have the potential to continuously
impact on the degree of fit. Hence, the evaluation
process must iteratively assess any changes in the
degree of fit throughout the various stages of the
project. This is achieved in three ways, through:
251
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
(1) screening the environment for possible
problems or solutions to problems that may
exist;
(2) assessment, comparison and selection of
appropriate options; and
(3) continuous learning and improvement.
As mentioned earlier the typical life cycle of these
projects consists of three major phases, namely
initiation, adoption and implementation. Within
these phases are a number of stages – agenda-
setting, matching, redefining/restructuring,
clarifying and routinising (Rogers, 1983)
(Figure 1). Evaluation for assessing organisation-
technology fit is therefore, conducted throughout
these phases.
Screening
Screening for possible problems or solutions
occurs at the agenda-setting stage of the project.
Evaluation is driven either from the perceived need
for a solution to a specific business problem or
from the awareness of a potentially valuable new IS
product or service. In the problem-focused case,
the first target of evaluation is likely to be the
development of a clearer understanding of the
problem. Thereupon, an active search for
solutions can be undertaken and the focus of
evaluation will then move to the development of
“awareness knowledge”, i.e. “principles
knowledge” and “how-to knowledge” for each
potential solution (Rogers, 1983).
In the technically-focused case, awareness of a
new IS product or service may occur in one of a
number of ways: through routine monitoring of
technological advances; as a result of publicity/
contact from potential providers of a particular
product or service; through monitoring of
competitors; or as a result of routine professional
contact. Once relevant business problems and
potential solutions have been identified, the focus
then moves to identifying and developing an
understanding of relevant business problems that
the product or service may address. In both cases,
the information gathered may be used to justify
funding for the next stage of the project, possibly
allowing for trialing of the most promising
solutions. Evaluation is likely to be iterative, with
the emphasis shifting between internal and
external foci as knowledge of both business
problems and potential solutions increases.
Assessment, comparison and selection of
options
At the matching stage the focus is on information
gathering. In the problem-focused case, a number
of possible solutions may be matched against the
business problem. In the technically-focused case,
potential solutions may be matched against a
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
number of problems to determine which, if any,
should be further investigated.
Evaluation will assess the feasibility of each
solution, its flexibilities and constraints, risks and
opportunities and any obstacles to
implementation. Stakeholder groups and potential
costs and benefits will need to be identified and
understood. The purpose of evaluation will be to
eliminate infeasible options and to compare
alternative viable options to select the best to take
forward to the adoption phase. Piloting can be
particularly valuable.
Although at a high level the IS product or
service to be implemented will have been selected
by the end of the initiation phase, it is likely that for
some sub-components of the project alternative
options remain. Thus, this type of evaluation is
likely to continue, but at a more detailed level, into
implementation. There may also be subsequent
re-evaluation of options in the light of experience
in use.
Continuous learning and improvement
As the project moves into the redefining/
restructuring stage of implementation, the fit
between organization and technology continues to
be evaluated and developed. Information is
actively sought as more “principles” and “how-to”
knowledge is acquired through experience in use.
Not only may the IS product or service be adapted
to better fit the organization, but the structure of
the organization and its ways of working may have
to be changed to accommodate the new product or
service: evaluation can be thought of as a feedback
mechanism between use and design. If significant
gaps in fit are found (these could arise through
changes either internal or external to the project or
be a result of unanticipated problems experienced
in the field), it may be necessary to re-evaluate the
options.
At the clarifying stage, as the product or service
is put into wider use, the fit between organization
and technology should be well defined, although
there may be minor improvements to be made.
The focus moves to supporting large-scale use.
Any misunderstandings, unwanted side effects, or
barriers need to be quickly identified and assessed
so that corrective action can be taken.
At the routinising stage, the product or service is
in wide use within the organization. By this stage,
evaluation for the purpose of developing
organization-technology fit will have effectively
ceased.
Justification and decision-making
Much of the information gathered in the
development of organization-technology fit will be
252
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
a source of input to project justification and
decision-making. This is an ongoing process
recognising that the information upon which
decisions are made will change over time.
In the agenda-setting and matching stages, prior
to any decision to adopt a particular solution,
decisions to experiment with or trial particular
options will be made on limited information.
Justification for a particular course of action may
be primarily based on information buying or risk
reduction at relatively low cost. As the project
progresses, however, and more information
becomes available this will be used to develop the
business case.
By the end of the initiation phase, all necessary
information should have been gathered to support
the decision on adoption of an IS product or
service, to plan for its initial implementation and to
justify the necessary funding. The primary purpose
of evaluation at this point is to provide the
necessary support to management to enable them
to take the adoption decision. This is not
necessarily a straightforward choice between
adoption and rejection of the IS product or service
for several reasons.
.
Although one course of action may be
preferred and recommended, alternatives may
need to be considered.
.
It may be felt that the time is not right to move
forward into implementation. This may be
because more information is needed or
because the plans for implementation are not
yet sufficiently developed, i.e. the project
effectively moves back to initiation.
Alternatively, if the products being considered
are undergoing rapid changes, a “wait and
see” strategy may be better.
.
There may be choices in scale of
implementation – too large and the project
can become difficult to manage and control;
too small and the full benefits will not be
realised.
The timing of the adoption decision is critical to
success. If it is taken too early, with insufficient
knowledge, there is a risk that the product or
service will be misused and will consequently, fail
leading to disenchantment and loss of confidence.
If the decision is taken too late, the costs will be
higher and the achievement of potential benefits
delayed.
An important difference between clinical trial IS
projects and conventional IS projects is that
adoption decisions in clinical trial projects are
likely to be subject to subsequent confirmation.
This may be a pre-planned review, a response to
changing circumstances external to the project or
as a result of problems during implementation. As
a consequence, evaluation will continue for the
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
purpose of justification of the project. As the
project moves through the implementation phase,
performance targets may be set and a performance
measurement programme introduced. As
experience grows and evidence is gathered, earlier
assessments of costs and benefits will be refined to
strengthen the business case.
Project monitoring and control
Evaluation for the purpose of monitoring and
control of the project will start in the matching
stage, as experimentation with or trialing of
potential solutions is undertaken and associated
costs start to be incurred. An important aspect of
evaluation at this stage is to ensure that the specific
objectives of experimentation are met and that the
necessary principles knowledge is gathered.
At the redefining/restructuring stage, monitoring
and managing the implementation process is
particularly critical: making sure that information
is communicated to the right people, that
processes and working practices are correctly
followed, technical aspects run smoothly, etc.
Performance targets and measurement
programmes may be instigated.
At the clarifying stage, the focus moves to
monitoring performance targets and supporting
the management of large-scale use. Any unwanted
side effects or barriers to meeting targets need to
be quickly identified and assessed.
At the routinising stage, the product or service is
in wide use within the organization and the project
will be drawing to a close. Once the project has
been completed, a post-implementation review is
likely to be conducted. As with conventional IS
projects, this will be primarily concerned with
product and process review (Earl, 1989).
An illustrative example: electronic data
capture for pharmaceutical clinical trials
To illustrate the IS evaluation life cycle for clinical
trial projects we present a case study carried out in
Glaxo Wellcome Research and Development
(GWRD) and, prior to the merger in 1995 of the
Glaxo Group with Wellcome plc, in Glaxo Research
and Development. (Dymoke-Bradshaw, 1999)
The case study describes the electronic data
capture (EDC) project, a major project to
implement EDC tools in clinical trials. Clinical
trials data have traditionally been and still are
largely collected on paper, using case record forms
(CRFs), which are completed by the investigator,
or diary cards, which are usually completed by the
253
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
patient. Speeding up the clinical trials process is a
key goal for most pharmaceutical companies and
the use of EDC is seen as a potential way of
achieving this goal.
The start of the project can be traced back to
1993 in Glaxo Research and Development, prior
to the merger with Wellcome, when a small
Clinical Data Capture Strategy team was set-up in
the UK Medical Operations Division to review
options for electronic data capture.
Overall, 14 major evaluation exercises were
identified in the EDC project and these have been
mapped against their evaluation purpose as shown
in Figure 2.
Organization-technology fit
Of the major evaluation exercises identified, nine
were primarily concerned with the evaluation of
organization-technology fit. Although in most
cases the information gathered was also used to
support evaluation for other purposes, this still
gives a clear indication of the importance of this
kind of evaluation.
Screening
In 1993, one of the first actions of the Clinical
Data Capture Strategy team was to commission an
in-house review of emerging technologies. The
purpose of this was two fold: first, to identify
available options and, secondly, to gain an insight
into the way other companies were implementing
these technologies. Subsequently, the team focused
on fax technology and combined CRF and data
entry screen design as areas offering most potential
for significant business benefit. One electronic
CRF (eCRF) product, based on a pen-computer,
was demonstrated but it was felt that the
technologies used, in particular handwriting
recognition, were not sufficiently mature at that
time. The team reported their findings in early 1994
and recommended that implementation proceed
only for fax-based CRF return, but that a small pilot
of an eCRF system can also be considered. A pilot
implementation of the pen-computer based system
was subsequently approved.
The initial review was driven primarily by the
belief that the capture and entry of clinical data
was the main remaining target for potentially
useful exploitation of IT, given that IS supporting
data processing and reporting for clinical trials
were already well established. There was also an
awareness that the majority of other major
pharmaceutical companies were already active in
the area of clinical data capture and that a number
of commercial products capable of conferring
significant business benefit were becoming
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
Figure 2 Mapping of EDC evaluation exercises against evaluation purpose
available. The purpose of the exercise was “to
review options and to produce a proposal for the
integrated use of appropriate technologies” within
a time frame of 5 months, to fit in with the
organization’s internal budget cycle. This involved
identifying and developing awareness knowledge
of potential solutions, either commercially-
available products, in-house development, or
collaborative ventures with other pharmaceutical
companies. An important component of the review
was understanding how other companies were
using these technologies and learning from their
experiences.
It is interesting to note that by August 1995,
when interviews for this case study commenced,
attention was clearly focussed on eCRF systems,
which were seen as offering the ability to shorten
clinical development programmes by several
months. Fax technology and combined CRF and
data entry screen design, which had originally been
given a high profile, had not been as successful as
anticipated and were no longer being actively
pursued.
Assessment, comparison and selection of
options
Four major evaluation exercises were initially
conducted for the purpose of assessing, comparing
254
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
and selecting options within the project: the
evaluation of pilot use of the pen-based eCRF
system previously identified, the piloting of a
number of other, alternative eCRF products, the
review of eCRF experience arising from the pilots,
and the selection of the hardware service provider.
The clinical trial chosen for the pen-based
eCRF pilot was conducted during the second half
of 1994 in two European countries at 13
investigator sites. Although the particular system
chosen for the pilot was unsuitable for use as
piloted, the feasibility and potential benefits of
eCRF systems in principle were clearly
demonstrated.
The objectives of the pen-based eCRF pilot
were again focused on information gathering:
.
to identify by how much the technology could
shorten time to market;
.
to assess the effects of the technology on
working practices;
.
to determine the acceptability of the
technology in the clinic; and
.
to consider options for pen-based CRF
processing and their associated costs and
benefits.
The second evaluation exercise, piloting a
number of alternative eCRF products, started in
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
April 1995. Products from three different vendors
were used in four clinical trials, ranging from a
short, single-site study in the US to a major,
18-month long clinical trial conducted in 50
centres in six European countries. The main
objective at this stage was to determine how to
select and scale up on one eCRF system for
“production use” (the term used by the EDC
project team for routine use within the
organization).
In October 1995, a major review of eCRF
experience to date was held. The primary objective
of this meeting, which drew heavily on the
information gathered from the previous pilots, was
to select one or more preferred eCRF software
vendor for production use of EDC. Other
objectives included the development of scenarios
for working processes and use of technology,
determining implementation plans, considering
the further evaluation of new technology
(especially for non-CRF data) and reviewing the
key lessons from the pilots. The review was held
jointly with the US and UK representatives and a
great deal of time was spent ensuring that there
was a common view of the technology within the
team through sharing information and agreeing
selection criteria. Following the review, a contract
for the provision of software and software services
was negotiated with the selected eCRF software
vendor and finalised in May 1996, when funding
for the next 2 years of the project was approved.
The process of selecting a hardware service
provider was also initiated, following the review.
Initially, the service requirements were defined at a
fairly high-level, but these were specified in greater
detail as the selection process progressed and
understanding increased through discussion, both
within the EDC team and with the potential
service providers. The final selection was made in
August 1996 and the contract was signed in
November 1996.
Finally, there was one additional evaluation
exercise, the identification and assessment of
electronic systems for the capture of data recorded
directly by the patient (diary cards, quality of life
questionnaires, etc.). This was conducted in a
sub-component of the project determining EDC
options for non-CRF data. This was carried out in
the first half of 1997, when the implementation of
eCRFs was well under way, with the objectives of
defining requirements for different types of diary
cards and questionnaires and of identifying the
most promising potential solutions.
Continuous learning and improvement
In late 1996, as the EDC project moved into
production use, a programme for the routine
collection of milestone and workload metrics and
255
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
user feedback from eCRF studies was set-up and
data were collected on an ongoing basis for more
than a year. Experience was also captured through
informal communication between members of the
EDC team and the project teams. Two metrics
reviews were conducted, one in mid 1997 and a
second at the end of that year, which drew on and
summarised the evaluation data. These exercises
were all strongly focused on learning for
improvement.
Objectives for the routine collection of metrics
and user feedback included measuring the impact
of eCRFs on clinical trial timelines and resources,
collecting feedback from various stakeholders and
making recommendations for improvement.
The objectives of the first metrics review were to
take a critical look at the current status of eCRF
implementation, to demonstrate the successes and
to identify the issues being addressed with the
overall aim of providing a baseline against which
improvement in the use of eCRFs could be
measured over time. The second review had similar
objectives, concentrating on changes since the first
review. Standard spreadsheets were used to collect
cycle time and resource metrics from clinical trials
project teams, together with interviews with project
team members to gather “soft” metrics.
Questionnaires were used to collect feedback from
clinical trial monitors working in the field and from
clinical investigators and study nurses.
Later, as the EDC transition team was planning
for the transfer of the eCRF system into routine
use, helpdesk and hardware and software service
provision were reviewed with the objective of
revising the services to match the current needs,
which had changed significantly since they were
first implemented.
Project justification and decision-making
An incremental approach was taken for the
justification of the EDC project. Much of the
supporting information came from the evaluation
exercises already described for the development of
organization-technology fit. In the initiation phase
of the project, justification of the early fax and pen-
computer pilots and the subsequent pilots of
alternative eCRF products was primarily an “act of
faith”, recognising that the necessary information
was not yet available to build a solid business case.
In applying funding for the pilots, the emphasis
was primarily on estimating costs, not benefits.
Although some potential benefits were identified,
it was not possible to quantify them and an
important objective of the pilots was to acquire the
information and understanding necessary to
develop the business case.
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
The adoption of the selected eCRF product for
large-scale implementation was based on a capital
expenditure proposal which was reviewed and
approved through the standard internal GWRD
mechanisms in May 1996. The proposal was
supported by a cost-benefit analysis based on
reduced clinical development time and hence,
earlier regulatory submission and ultimately,
product launch.
At the end of 1996, EDC became a component
project within core process redesign (CPR), a
major medical operations process improvement
project. This transfer of responsibility prompted a
review of EDC within the broader context of the
redesign framework. The project justification was
reconsidered and the validity of implementing the
eCRF system was re-examined and confirmed.
The routine evaluation of the early eCRF trials
played an important part in the review by
providing evidence showing that expectations were
being met and an active programme for continuous
improvement was in place. CPR placed particular
emphasis on the need for a stronger cost-benefit
case as the business climate within GWRD
changed.
Over 1997, the senior management view of the
eCRF system in use changed as experience grew
and its limitations were recognised. The strategic
focus within the organization moved to “next
generation” EDC and the investigation of future
EDC solutions became a separate project. The
primary focus of the EDC project became the
implementation of the selected eCRF system as a
tactical, short-term tool and as a way of building
skills and knowledge in preparation for the piloting
and subsequent scaling up of future EDC
technologies. Implementation proceeded steadily
over the year and by December 1997, the eCRF
solution was implemented between 10 and 15
percent of clinical studies.
In parallel with the implementation activities,
an EDC transition team was set-up in May 1997
to make recommendations on how the
management and use of the selected eCRF
solution should move from the EDC project
team to integration within the Glaxo Wellcome
line functions. The team re-evaluated the eCRF
business case and extended the metrics collection
to develop a clearer understanding of the costs
and benefits incurred in using the eCRF system
compared to paper-based studies and contract
research organizations. The team also developed
tools that could be used to support decision-
making on the use of eCRFs for future studies,
drawing on the re-evaluation and on earlier work
on eCRF study selection criteria.
To summarise, evaluation for justification and
decision-making occurred throughout the life
256
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
cycle of the project; as more information became
available, understanding of the implications of
EDC increased and the business case developed to
reflect this.
Project monitoring and control
Many of the evaluation exercises concerned with
the development of organization-technology fit also
provided information which was used to support
project monitoring and control. This was
particularly true of the routine collection of
standard metrics, which included data on
resources, time lines and costs incurred as well as on
the realisation of benefits which could be compared
with earlier estimates and used to monitor project
progress. By February 1998, when the last case
study data were collected, the project team was
being disbanded and the recruitment of staff for the
new organizational structures was under way.
Evaluation was planned to continue with the
objectives of routine performance monitoring and
providing information and support for decision-
making on data handling strategies.
Postscript – an update on the use of EDC
at Glaxo Wellcome 1998-2002
The level of use of the selected EDC system has
not increased significantly from 1997, when the
case study described in this paper was completed.
Take-up within the organization has been patchy –
some project teams have made heavy use of the
technology whilst others have remained with
paper-based studies. Some newer, Web-based
technologies were used for one or two clinical
trials, but did not replace the original system.
When Glaxo Welcome merged with
SmithKlineBeecham (SB) in 2001 to form
GlaxoSmithKline (GSK), the original GWRD
EDC system was nearing the end of its life and the
organization decided to develop and consolidate
on an integrated version of the former SB in-house
system, RDMi. The original GWRD system was
finally decommissioned in late 2001. At the time of
writing, in January 2003, EDC continues to be a
key component of GSK’s clinical data
management strategy.
Conclusion
The evaluation of clinical trial projects requires a
different set of methods to those used for
conventional projects. This requirement stems
 An integrative, iterative approach to IS evaluation
Catherine Dymoke-Bradshaw and Benita Cox
from the fact that the underlying project processes
are different.
A critical point about IS projects for clinical
trials is that often a full understanding of
requirements can only be developed through
practical experience gained as the project evolves –
the project process is complex and iterative and IS
evaluation is an integral part thereof. Further, the
underlying technology may be changing and this,
together with any opportunities for influencing the
direction of change, needs to be understood and
accommodated.
The development of organization-technology fit
lies at the core of these projects. Evaluation for this
purpose can be thought of as a coupling
mechanism between use and design and as a tool
for continuous learning and improvement. It also
provides information to support evaluation for the
traditional purposes of project justification and
project control. The importance of evaluating the
fit between organization and technology on an
ongoing basis from project initiation through to
routine operation is clearly illustrated in the case of
Glaxo Wellcome’s EDC project. Of the 14 major
evaluation exercises identified in the project, nine
were primarily conducted for this purpose.
This leads us to the conclusion that in
developing methods for evaluation of IS projects in
pharmaceutical clinical trials there is a need to
draw more extensively on the established body of
academic literature in the field of innovation.
References
Argyris, C. and Schon, D. (1996), Organizational Learning II:
Theory, Method and Practice, Addison-Wesley, Reading,
MA.
Baker, B. (1995), “The role of feedback in assessing information
systems planning effectiveness”, Journal of Strategic
Information Systems, Vol. 4 No. 1, pp. 61-80.
Clemons, E. (1991), “Evaluation of strategic investments in
information technology”, Communications of the ACM,
Vol. 43 No. 1, pp. 23-36.
Dymoke-Bradshaw, C. (1999), Evaluation in Information Systems
Innovation Projects, Imperial College, London.
Earl, M.J. (1989), Management Strategies for Information
Technology, Prentice-Hall, London.
Edzerodt, P. and Madsen, K. (1988), “Information systems
assessment as a learning process”, Information Systems
Assessment: Issues and Challenges, Elsevier, Amsterdam.
Farbey, B., Land, F. and Targett, D. (1992), “Evaluating
investments in IT”, Journal of Information Technology,
Vol. 7 No. 3, pp. 109-22.
257
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 249–257
Farbey, B., Land, F. and Targett, D. (1993), How to Assess your IT
Investment. A Study of Methods and Practice,
Butterworth-Heinemann, Oxford.
Farbey, B., Land, F. and Targett, D. (1995), “A taxonomy of
information systems applications: the benefits evaluation
ladder”, European Journal of Information Systems, No. 4,
pp. 41-50.
Galliers, R. (1991), “Strategic information systems planning:
myths, reality and guidelines for successful
implementation”, European Journal of Information
Systems, Vol. 1 No. 1, pp. 55-64.
Gregory, A. and Jackson, M. (1992), “Evaluation methodologies:
a system for use”, Journal of Operational Research
Society, Vol. 43 No. 1, pp. 19-28.
Hawgood, J. and Land, F. (1988), “A multivalent approach for
information systems assessment”, in Bjorn-Anderson, N.
and Davis, G. (Eds), Information Systems Assessment:
Issues and Challenges, North Holland, Amsterdam,
pp. 103-24.
Hochstrasser, B. and Griffiths, C. (1991), Controlling IT
investment: Strategy and Management, Chapman & Hall,
London.
Kumar, K. (1990), “Post implementation evaluation of computer-
based information systems: current practices”,
Communications of the ACM, Vol. 33 No. 2, pp. 203-12.
Orlikowski, W. (1992), “The duality of technology: rethinking the
concept of technology in organizations”, Organization
Science, Vol. 3 No. 3, pp. 398-427.
Powell, P. (1992), “Information technology evaluation: is it
different?”, Journal of the Operational Research Society,
Vol. 43 No. 1, pp. 29-42.
Remenyi, D., Money, A. and Twite, A. (1993), A Guide to
Measuring and Managing IT Benefits, NCC Blackwell,
Oxford.
Remenyi, D., Sherwood-Smith, M. and White, T. (1997),
Achieving Maximum Value From Information Systems,
Wiley, Chichester.
Rogers, E.M. (1983), Diffusion of Innovations, 3rd ed., The Free
Press, New York, NY.
Serafeimidis, V. and Smithson, S. (2000), “Information systems
evaluation in practice: a case study of organizational
change”, Journal of Information Technology, Vol. 15 No. 2.
Symons, V. (1993), “Evaluation and the failure of control:
information systems development in the processing
company”, Accounting, Management and Information
Technology, Vol. 3 No. 1, pp. 51-76.
Targett, D. (1996), Keynote Address, Third European conference
on the evaluation of information technology, University of
Bath.
Walsham, G. (1993), Interpreting Information Systems in
Organizations, Wiley, Chichester.
Willcocks, L. (1994), Information Management. The Evaluation of
Information Systems Investments, Chapman & Hall,
London.
Willcocks, L. and Graeser, V. (2001), Delivering IT and e-business
Value, Butterworth-Heinemann, Boston, MA.

Defining ICT proposals
Fred J. Heemstra and
Rob J. Kusters
The authors
Fred J. Heemstra and Rob J. Kusters are Professors based at
the Open University, The Netherlands.
Keywords
Cost benefit analysis, Investment appraisal,
Communication technologies
Abstract
The paper presents an approach for defining ICT investment
proposals. The approach was developed on the authority of the
Dutch Ministry of Transport, Public Works and Water
Management. The Ministry was confronted with a growing
amount of ICT investment proposals. Since there was no format
for describing a proposal and no procedure for obtaining the
data on the basis of which a proposal was described, the
decision to select proposals for funding missed any kind of
rationality. Proposals differed enormously and hence were
incomparable. The evaluation of existing ICT assessment
methods showed that there was not any method that would
meet the expectations and demands of the Ministry. The main
points of criticism concentrated on the limitation of applying a
fixed set of data for describing a project proposal and the
absence of a process to support the selection of a meaningful set
of data to describe a project. Inspired by the multi-criteria score
method of Parker a tailor-made approach for defining ICT
investment proposals was developed, taking into account the
characteristics of the governmental organization. In this paper,
the approach as well as the developmental process with regard
to this approach are presented. The overall conclusion is that a
tailor-made approach, which is developed with the full
cooperation of the user organization, has considerable
advantages since local knowledge and expertise is incorporated
in this type of approach.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 258–268
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548661
258
1. Definition of the problem and research
questions
The research described in this paper was
commissioned by the Dutch Ministry of
Transport, Public Works and Water Management
(DMT). This Ministry is one of the largest
ministries in The Netherlands with regard to its
number of employees and budget. The
organisation is characterised by a decentralised
structure with local and regional offices, which on
the one hand have a high-level of autonomy and on
the other hand depend on the central headquarters
for funding (larger) projects. In many projects the
information and communication technology
(ICT) is an important component. Each year tens
of millions of euros are invested in ICT. This
amount pales beside the total number of all ICT
investment proposals that are submitted to the
central headquarters by the decentralised offices
each year for acceptance and funding.
Regarding the substantial sums of money
in ICT investments and the necessity to spend
the (relatively) limited budget as optimally as
possible, it is vital to have a good mechanism
for defining and selecting ICT investment
proposals for the DMT. However, in practice it
is the case that:
(1) DMT has no guidelines how ICT proposals
submitted by decentralised offices must be
defined, described, presented, etc.;
(2) as a consequence proposals differ enormously
with regard to the applied format, level of
detail, language/vocabulary used, etc.; and
(3) the decision makers at the headquarters are
confronted with a situation in which
.
a choice has to be made between
incomparable proposals,
.
no criteria exist for making an objective
choice, and
. no guidelines/procedures are available to
support the selection process.
It is in this situation that the research presented
here took place. Research focused on a part of the
cost-benefit analysis, which consists of the
following elements (Renkema, 1999).
(1) Definition phase. Defining a proposal:
.
customizing the approach,
.
gathering and analysing data required to
describe a proposal,
.
assessing the data, and
.
achieving consensus on results.
(2) Assessing phase. Selecting proposals:
.
achieving consensus on selection criteria
and weighing factors,
.
comparing different proposals,
.
ranking proposals, and
.
selection.
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
We focused on the definition phase. In this phase,
we can identify two relevant issues. The first issue
deals with data. If the top management wants to
base its decisions on the data described in the
project proposal, the proposals should be
comparable. This can be achieved when all
proposals are required to apply an identical format.
It is not so much the use of an identical proposal
format layout that is essential, but rather the use of
identical data sets based on published and
accepted (within the DMT) definitions.
Therefore, a key element of the design is the
decision as to what data should be collected to
provide a description of the proposed
development. This data should be sufficiently
comprehensive to capture the costs and benefits
while at the same time allowing comparison
between proposals. On the basis of this we can
formulate the first research question as follows:
Design a data set that results in a sufficiently
complete ICT-project proposal.
The second issue deals with process. This aspect of
completeness is not sufficient in itself. If the top
management wants to be able to base its decisions
on the data described in the project proposal, they
will need some kind of assurance that the data
presented are correct in the sense that the data
offers a reliable picture of the actual available
knowledge. The mere fact that a manager requires
an ICT investment proposal to be submitted
according to a prescribed format does not imply
that the data collected and assessed in this way are
reliable. Wholehearted support from professionals
is unlikely to be obtained by invoking authority
(Mintzberg, 1983). This led to the realisation that
a reliable and accepted process for gathering and
selecting data was required. This process should
match the organizational language, culture and
habits and way of working and thinking. On the
basis of this we can formulate the second research
question:
Design a data collection process that will provide
sufficiently reliable data for ICT-project proposals.
The approach taken to answer these design
questions was as follows. The research was aimed
at deriving a workable solution for DMT,
considering its specific situation and requirements.
First of all, an extensive requirement study was
carried out. This study focused on two areas. An
obvious first field for study is that of ICT
evaluation. An extensive body of literature in this
field is available and this provided valuable insights
into the data issue. For the process issue we
decided to rely on literature describing the subject
of processing human information, since their
expertise in designing data collection and
evaluation processes can be found. Next, by
259
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
applying these results and taking the specific
circumstances and requirements of DMT into
account we obtained requirements for the design
of a DMT-specific approach and used them to
design both data and process. The survey and its
conclusions are presented in Section 2. The
resulting design is presented in Section 3. The
design evaluation is presented in Section 4.
The results obtained are specifically applicable
to DMT. However, we could also identify a
number of more generally applicable lessons when
reviewing the research in hindsight. These are
presented in Section 5.
2. Design requirements
In this section, we will identify design
requirements that will lead to the design of a data
set and data collection process that is specific for
the DMT. With regard to the data set we first
examined relevant literature in the field of the cost-
benefit analysis. A short survey of this literature
will be presented in Section 2.1. In Section 2.2, we
draw a number of conclusions for the DMT
setting.
Next, we identified requirements regarding the
process of collecting and assessing data for an ICT
investment proposal. This process can be seen as a
specific instance of a human information-
processing activity. This is a research field that has
been studied in depth and in which practical
results are available. So for the purpose of this
study, rather than exploring the process aspects
discussed in the literature on cost-benefit analysis,
we decided to return to the original source of
information. Section 2.3 includes a brief overview
of relevant literature in this field, followed by a
number of identifiable requirements in Section
2.4. Section 2.5 summarises the resulting
requirements.
2.1 Approaches towards an IT cost-benefit
analysis
Many methods and techniques have been
proposed for assessing ICT investment proposals.
As early as 1961, the International Federation of
Information Processing devoted its first conference
to evaluation issues (Frielink, 1961) and Joslin
(1986) wrote a book on computer selection.
Ever since a vast amount of literature has been
produced on the subject. For example, Renkema
and Berghout (1997) identified 67 different
methods for cost-benefit analysis, which can be
distinguished as follows.
.
The traditional financial methods (Brealy and
Myers, 1988), completed by the balanced
scorecard (Kaplan, 1986) and/or the option
theory (Renkema, 1999).
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
.
The multi-criteria score methods (Parker et al.,
1989, Ward and Peppard, 2002). The
approach described in the paper can be
considered as an example of the multi-criteria
score method.
. The portfolio methods (Bedell, 1985).
.
The ratio methods (Strassman, 1990).
2.2 Design requirements derived from
current approaches to an IT cost-benefit
analysis
Defining ICT proposals requires agreement on the
data that, when combined, describe a proposal.
In most assessment methods these data are
defined as criteria. We feel that data/criteria
form the basis for defining ICT project
proposals and they are therefore of crucial
importance. When exploring current approaches
on identifying relevant data, we drew the following
conclusions.
. Most methods have a tendency to apply a
fixed set of data.
.
However, there is no agreement as to what
data should be applied (Irani, 2002).
.
There is a general agreement on the
importance of financial criteria, however,
non-financial criteria also play an important
role in many approaches (Ward and Peppard,
2002).
.
The scope covered by the data varies widely,
with a substantial number of approaches only
taking money into account (Brealy and Myers,
1988), while others also take other aspects
into account (Irani, 2002; Parker et al., 1989;
Willcocks, 1994).
.
No solid theoretical foundation is provided for
the selection of (the set of) data.
Based on these findings we concluded that there is
no winner among the many advocated methods on
the basis of theoretical considerations. This
prompted us to take the following design decisions:
.
select the best type of approach on offer, and
.
take this as a starting-point, but accept that
changes will probably be needed to match the
specific DMT circumstances.
In selecting the “best type of approach” we applied
the criteria:
.
of acceptance in practice, as shown by the
approach used in practice either under its own
name or disguised as a proprietary method,
.
of acceptance in theory, as shown by the
number of references made,
. of clearness, as demonstrated by an
understandable description, and
.
finally, the readiness to which this approach
leant itself to adaptation to local
circumstances (specific to DMT).
260
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
After some discussion this led to the selection of
the data set described by Parker et al. (1989) as the
basis for the data set specific to DMT.
2.3 Defining ICT proposals: a human
information-processing activity
Processing information is a difficult task. In this
section, we will show arguments to support the
assumption that a team has more information/
knowledge at its disposal than an individual.
Defining ICT and, more specifically, assessing the
defined data should therefore, be carried out by a
team rather than by an individual.
This assumption will be further examined in
this section, by using a general information-
processing model from the field of social
psychology, which shows a number of major
limitations on individual information processing,
arguing that teams or groups may remedy such
limitations. The model presents a number of
agreed conditions for group effectiveness
and efficiency for carrying out processing
information, and translating such conditions
into specifications for the practical design of
instruments and procedures to be used in defining
ICT proposals.
2.3.1 Individual information processing: perception
An individual who values the data in a proposal,
forms a judgement on the basis of what he sees, in
combination with what he (or she) knows, or
rather: he combines what he thinks and sees with
that what he thinks and knows With this we mean
that “seeing data” is not just a simple observation,
but the outcome of a process of perception, which
goes through a number of stages. Figure 1 shows
such a model as described by Janis et al. (1999).
This model implies that two observers of the
same proposal are likely to form different
judgements with regard to the data that define the
proposal, due to:
.
selective focus on people, events and objects;
.
the interpretation of what is perceived; and
.
selective memory, which is retained in
memory
Figure 1 The perception model
 Defining ICT proposals The Journal of Enterprise Information Management
Fred J. Heemstra and Rob J. Kusters Volume 17 · Number 4 · 2004 · 258–268

People focus selectively on stimuli, which are interaction of specialized individuals who share
congruent with their needs, and these depend in their knowledge, by turning private knowledge into
turn on position, role and tasks. People in different public knowledge and tacit knowledge into explicit
functions or on different levels in the hierarchy knowledge (Nonaka and Takeuchi, 1995). This
may select different stimuli that could later be also applies to defining ICT proposals, but this
interpreted as “risky” or “costly”. does not make it an easy task for a group. In order
People have different schemata to order the for groups to outperform individuals, we must be
stimuli on which they selectively focus, influenced able to use the advantages of groups by creating the
by personality and prior socialisation (education, necessary conditions for effectiveness and
training, group membership). By hiring people efficiency of group performance for the specific
from specific backgrounds and organising them in type of task.
groups, the organization reinforces differential
interpretation of data by different members.
2.4 Design requirements for group
What is used of these interpretations
effectiveness and efficiency
constructed by persons on the bases of their
The effectiveness of a group can only be measured
selective attention to stimuli depends on what they
in terms of a specified task. In our case this task
selectively store and retrieve from their memories.
consists of the definition, analysis and valuation of
Selective memory and selective forgetting in
data in a proposal. What kind of task is this? It is
organizations is again dependent on positions,
certainly not a task in which the outcome consists
tasks and roles.
of the only “correct answers”. Neither is it a
Selective memory determines a selective focus
political task in which resolving conflicts of
on stimuli and differential interpretation. It is this
interests between the actors is central to the
self-reinforcing character of selective processes
process. However, the task involves the resolution
which makes human perception very vulnerable to
of substantive conflicts required to integrate
bias. Stimuli that are not attended to will not lead
knowledge from different backgrounds and
to interpretive schemata and not create memories
reconciling different frames of actors: it has
that will help to see such stimuli in the future!
characteristics of a so-called “cognitive conflict
Blindness tends to be self-perpetuating in
task” (McGrath, 1984).
individual perception (see the “jelly model of
The effectiveness of a group carrying out such a
retention”, inspired by De Bono as described in
task can be judged in terms of:
Weick (1979)). .
the degree in which the cognitive input of
This makes the responsibility of an individual for
participants has been used, and
the total cost-benefit analysis phase problematic. .
the degree of integration of this knowledge as a
How can we guarantee that bias with regard to the
result of group discussion
assessment and valuation of data is excluded? This
would require access to a very wide range of stimuli, We call a group efficient when it is effective with
openness towards different interpretational relatively few resources in terms of participant time
schemata and a very strong memory. and effort.
A general model of group functioning
2.3.2 Defining ICT proposals as a group task (van Zanten, 1996) links group results with
It seems more reasonable therefore, to define the group processes and input (Figure 2).
tasks of defining proposal and data valuation as a
2.4.1 Input
group task. A group of individuals with a sufficient
First of all, the task definition must be clear for all
diversity in disciplinary backgrounds, position,
involved at the very start. This may involve a
roles and tasks, will counterbalance the perceptual
formal task assignment, briefing of new group
one-sidedness of individuals. A group can improve
members, etc. The composition of the group must
the phase of defining ICT proposals by:
be in line with this task. In our case, the variety of
.
paying attention to a larger range of potentially
backgrounds and the representation of all relevant
relevant stimuli, experienced by people in
different positions with various backgrounds
Figure 2 A general model of group functioning
and different tasks;
.
employing a larger collection of interpretative
schemata to the available range of stimuli; and
.
retaining a broader range of interpretations in
a (collective, shared) memory
What we are suggesting here, is one of the core
ideas in knowledge management: we can use the
group to create superior knowledge through the
261
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
knowledge must be warranted. The structure and
rules of the team must support the task. In our
case, where the task demands a free flow of
information and open communication, there
should be no hierarchical structure. The group
culture should support the requirements of the
task. One decision about the group structure is
leadership. Here a facilitator can play a role, as will
be discussed later. Group size is also a variable. In
our case diversity calls for relatively large groups,
but this should be balanced with efficiency: each
new member makes communication more
complex.
2.4.2 Process
The central processes in this type of group involve:
.
eliciting individual (private, maybe tacit)
knowledge from participants,
.
sharing this knowledge through
communication in the group, and
.
combining, integrating this knowledge
through group discussion.
These processes can be promoted by adequate
procedures and techniques:
.
procedures to “tap” knowledge from
participants prior to meetings,
.
techniques to summarize and present shared
knowledge to support group discussion, and
.
procedures to ensure the quality of group
discussion.
The fact that much knowledge is local, i.e. specific
to the context of an organization or a project,
implies that procedures to elicit, integrate, share
and discuss knowledge should be expected to be
effective and efficient insofar as they are designed
to deal with local knowledge: they should deal with
locally relevant themes and use the frames of
reference of the participants.
Procedures should be designed so as to
counteract the natural tendency of groups toward
inefficiency. For example, groups tend to block the
production of ideas by individuals. This may be a
reason to combine group work with individual
work.
The nature of the task demands that procedures
favour diversity in information and interpretations.
Actual group dynamics may negatively impinge
on group effectiveness and efficiency. One problem
frequently mentioned is that of the destruction of
the necessary (cognitive) variety in the group. This
is also known as groupthink (Janis et al., 1999).
Anxiety and pressures on individuals to conform
may lead to inferior reasoning in groups in difficult
situations. Patterns of defensive reasoning may
obstruct the necessary learning processes in groups
dealing with difficult cognitive tasks (Argyris,
1983, 1990). Win-lose dynamics and unilateral
262
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
control patterns must therefore, be corrected in
such groups.
Applying procedures may decrease the
likelihood of such negative group dynamics
occurring and promote discussion rules that favour
criticism and debate, like the devil’s advocate and
the dialectic method (Janis et al., 1999).
A facilitator, i.e. a person with the specialized
role of enabling the processes that lead to required
group outcomes, may be essential for maintaining
procedures for effective group work and correcting
possible negative group dynamics that may arise.
Facilitators may, among others, correct defensive
patterns and win-lose dynamics in groups,
guarantee the equal participation of all group
members or deal with emotional problems that
block effective functioning.
2.5 Summary of design requirements
The design requirements derived until now can be
divided into two categories as follows.
(1) Data requirements. No data set based on a
generic data set found in the literature could
be identified. It was decided to take the data
set suggested by Parker et al. (1989) as the
starting-point for a follow-up specific to
DMT.
(2) Process requirements. From a human
information-processing perspective more
conclusive theoretical support for process
design requirements can be identified. We
accepted these requirements and used them as
the starting-point for process design.
Requirements were identified in the following
areas:
.
clarity of task description,
.
team composition,
. structure and rules for team behaviour,
.
team culture,
.
use of a facilitator, and
.
procedures and techniques that support
eliciting individual knowledge, sharing
this, and knowledge by communication in
the group and combining and integrating
this knowledge through group discussion.
In Section 3, we will describe the way in which
these requirements were implemented.
3. The approach to defining ICT project
proposals
The theoretical design requirements identified in
the previous section were enhanced by
requirements specific to DMT in order to meet the
expectations and demands of the DMT. These
requirements were identified by:
 Defining ICT proposals The Journal of Enterprise Information Management
Fred J. Heemstra and Rob J. Kusters Volume 17 · Number 4 · 2004 · 258–268

.
interviews with key players in the DMT, Table I Non-financial criteria
.
studies of existing project proposals Proposed by Parker et al. (1989) Used by DMT
documentation, and
.
evaluations of existing ICT investment Risk factors
methods. Organizational risk Organizational risk
Specification risk Specification risk
The choice for the most suitable way of eliciting Complexity of field of application Complexity of field of application
requirements was based on the approach of Technical risk Technical risk
Davis (1982). He refers to several strategies for Infrastructure risk Infrastructure risk
determining requirements; three of these Non-financial benefits
requirements fit in with the approach to eliciting Strategic match Strategic match
requirements we used: “asking” (interviews), Management information Management information
“based on characteristics of their own Competitive advantage Political pressure
organisation” (studying documentation) and Competitive response Political information
“reference approach (studying existing Infrastructure match Infrastructure match
methods)”.
Based on the design requirements mentioned in Table II Developmental costs: staffing
Section 2, it was concluded that a tailor-made Internal staffing costs
approach to defining ICT project proposals had to Direct Indirect External
be developed for the DMT. In this section, we will Phase (in days) (in days) staffing costs
focus on what we think are the two most important
issues of the approach. Project preparation
Definition study
(1) Developing a support mechanism to select
Basic design
relevant data for defining a proposal and
Detailed design
estimating the value of the data.
Realisation
(2) Developing the process for defining ICT
Implementation
project proposals. Conversion
3.1 Selection of relevant data for defining Training
proposals Column total (days) –––
Based on the generic data set presented in Internal tariff (per day) X euro per day –––
Parker’s approach, a set of localised data was Total (euro)
defined Localisation took place in two areas.
The first area where adaptations were made was a number of representative project proposals it was
that of quantitative financial data. In fact a decided that the risks identified by Parker et al.
significant part of the effort went into were sufficiently applicable to the DMT situation.
structuring and detailing data on financial costs However, when examining non-financial criteria it
and benefits in such a way as to provide a was deemed necessary to make a number of
sufficiently detailed starting-point for estimating alterations. The issue of competition is not an
costs and benefits, while at the same time aspect that is relevant for DMT. On the
achieving a match with the current financial other hand, issues were added that referred to
reporting structure of DMT. This resulted in
defining detailed costs and earning tables, an Figure 3 Developmental costs: other
extensive system of checklists set-up in a local
language and linked to the existing registration
and accounting systems. Tables I and II are
examples of this. Overall 22 tables were required
to cover all relevant data. Financial data will
probably suffice when looking at applications
that further the efficiency of current processes.
If a more innovative application area is
suggested this will usually not suffice. To
identify systems where non-financial aspects
could be more important systems are
characterised as either innovative or supportive
(Figure 3).
The approach by Parker et al. also proposes
using specific qualitative data in order to describe
risks and non-financial benefits. Based on the
discussions with DMT stakeholders and a study of
263
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
political information, indicating that the proposed
system was put into effect to answer a request
for information from either government or
parliament, and political pressure, indicating that a
system has been put into effect as required by the
government. Table I shows the original proposal by
Parker and the choice made with regard to DMT.
All results are consolidated and are shown in
Figure 4.
3.2 Developing the process for defining ICT
project proposals
Based on the requirements identified in the
previous section a process for defining an ICT
project proposal was developed. We will limit our
discussion to the presentation of the basic steps of
this process. An extended description of the
process can be found in Heemstra and Kusters
(2000).
(1) Selecting the group of people. The goal of the first
step is the composition of a “proposal
definition team” that will perform the
activities of the definition phase of the ICT
selection approach. An important issue here is
insuring representation of each stakeholder
group and knowledge area involved in the
project by at least one agent.
(2) Explaining the approach and planning tasks.
The goal of step 2 is twofold. First of all it is
important to inform all stakeholders about the
work procedure. Secondly, it is necessary to
acquire commitment.
(3) Defining and assessing data. The process
facilitator interviews all people involved.
In this interview, each person’s opinion is
Figure 4 Consolidation form
264
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
requested using the (localised) data sets
introduced in Section 3.2.
(4) Analysis and consolidation of results. The
process facilitator consolidates the inputs
from all parties identifying points of
agreement and discussion. A report is sent to
all people involved.
(5) Defining and assessing collaborative data.
A general meeting is held based on the report
and chaired by the process facilitator The
objective of the meeting is to confront the
team members with each other’s perception of
the data assessment, to initiate a discussion on
differences in perception and, last but not
least, to agree upon the most likely data
values.
(6) Reporting. A final report is drawn up
containing the final and agreed ICT
investment proposal.
(7) Assessing the proposals. As mentioned
earlier, the paper deals with the definition
phase of the ICT selection process. We will
not explain the assessment phase of the
approach here.
3.3 From requirements to approach
When developing the ICT proposal selection
approach, we took care that all design
requirements were met. Table III summarizes how
the design requirements were implemented into
the approach.
4. Evaluation
The evaluation of the method consisted of three
parts:
(1) a so-called internal acceptance test,
(2) an application test, and
(3) an acceptance test.
In the remainder of this section the three tests will
be discussed briefly.
4.1 The internal acceptance test
After completion of the development it was tested
whether the approach met the requirements
postulated in the design. Table III was used as a
tool for validation and verification. In order to
guarantee an objective judgement an external
auditor supervised the internal acceptance test.
The test result showed that all requirements were
met satisfactorily.
4.2 The application test
The goal of the application was to find out if the
stakeholders could estimate the value of data
in the tables. In order to handle this general
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
Table III How requirements are used to develop the approach
Data requirements
Locally defined data set Using checklists based on generic data sets
from existing ICT assessment methods
Adapting these checklists to local circumstances
Process requirements
Task definition Clear design of the method coupled with an
explicit explanation of tasks and responsibilities
Group composition Selection of participants aimed at covering all
major roles and knowledge areas
Rules, structure Use of an independent facilitator with no political
ties to the project
Individual interview between participants and
facilitator as a basis for group discussion to
promote open communication
Group size to be kept reasonably low so as to
facilitate an efficient group meeting
Culture, leadership Use of an independent facilitator with no political
ties to the project
Elicitation Individual interviews as the basis for further
discussion to prevent blocking and groupthink
Use of a checklist specific to the type of project
in order to promote the motivation of participants
by showing relevance
Use of a checklist based on the jargon of the
organization in order to promote efficiency by
remaining within the frame of reference of the
participants, hence promoting understanding
Sharing All interview results are made available to all
participants
Interview results are used anonymously to prevent
pressures of having to conform
The facilitator also summarizes interview results in
order to provide a discussion list of potential risks
specific to the project
Combining Use of an independent facilitator to chair the
meeting in order to correct defensive patterns and
win-lose dynamics, ensure equal participation
and deal with emotional problems that block
effective functioning
Process guidance Use of an independent facilitator
Communication Sharing of knowledge and expertise in team
meetings
Discussion on relevant data in the jargon of the
organisation
question, the question was subdivided into five
simple questions as formulated in the caption of
Table IV.
During the application test three “real life”
projects were applied. All projects were in the stage
of defining a proposal. Instead of using their own
informal way of describing an ICT investment
proposal, the project participants were asked to use
the developed approach. This implied that the
steps 1-6 of the ICT selection process were
performed for three different project proposals and
that the checklists, containing the relevant data,
were completed for these projects. All in all, eight
265
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
stakeholders were involved in the application test:
the client, developer, project manager and project
consultant.
Table IV shows the format in which the results
were presented. As mentioned above, the table
caption refers to the evaluation questions; row 1
represents the 22 designed tables, while the
opinions/answers of the stakeholders with regard
to the questions are formulated in the cells (answer
options: yes, no, no answer). The row “Total”
counts the answers per question.
As can be concluded from Table IV (row
“Total”), the users were quite satisfied with the
usefulness of the tables. About 75 per cent of the
respondents judged the tables as significant and
only 11 per cent as insignificant. Despite this
positive feedback, 33 per cent had problems with
answering/completing the tables (too many, in our
opinion) and only 56 per cent regarded the
documentation as sufficiently clear. These two
negative aspects resulted in many suggestions for
improvement. On the basis of these results the
tables and documentation were adapted.
4.3 The acceptance test
After correcting the tables and documentation a
formal acceptance was performed. The
participants of the application test were
interviewed through a questionnaire. The
participants were familiar with the approach
towards defining ICT project proposals after
having applied this approach during the
application test.
The acceptance showed that all people involved
were generally positive about the approach,
specifically about the extensive use of practicable
tables, the interviews, which yielded more
productive data than through a written collection
of data, and the group meeting, which made it
possible to discuss issues in a sound way.
Respondents felt that applying the method was
successful in giving them a greater “grip” on the
project. They mentioned the following strong
features of the method:
.
the group meetings,
.
the surfacing of unexpected data,
.
the resultant overview of costs, benefits, and
risks,
.
an increased understanding of the proposed
project, and
.
an increased objectivity due to the presence of
the process facilitator.
They also mentioned the following weak points:
.
initial problems with an abstract formulation
of the data,
.
labour intensiveness, and
.
the process facilitator being unfamiliar with
the project.
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
Table IV Results of the application test
Question 2
Question 1 Problems with-
Table significant? answering?
Yes No N.a. Yes No N.a.
Table 1 7 0 1 0 6 2
Table 2A 8 0 0 5 3 0
Etc.
Total (per cent) 75 11 14 33 43 24
The respondents were divided on the issue of the
process facilitator. Some were of the opinion
that the support offered by a neutral outsider
(outside the project, but not necessarily outside the
organisation) could just as well be offered by
the project champion. Others stated that the
neutrality and focus of the outsider process
facilitator on the process itself provided a valuable
contribution. The authors support the latter view,
also based on additional evidence from other areas
such as risk management (Heemstra and Kusters,
1996).
The acceptance test resulted in a number
of minor adaptations. Finally, the approach
was implemented at the Ministry of Transport,
Public Works and Water Management as a
standard procedure for selecting ICT proposals.
5. Lessons learned
The first lesson we derived from this study deals
with the data. As mentioned earlier, we concluded
that current methods did not provide sufficient
support to answer the question “what data should
be used in a specific situation”. We selected a well-
known approach, that of Parker and Benson,
which was considered (not only by us) to be the
“best of the breed”. However, we found the
following.
(1) Much work was needed to (re-) define
financial data in such a way as to match the
reporting practices and language and jargon
used in the specific DMT setting; and
(2) The non-financial benefits were only partially
applicable in the specific DMT setting. Some
criteria were irrelevant while other relevant
criteria had to be added.
A traditional answer to the question “what data are
relevant for deciding which project proposal to
fund”, is money. However, this answer is no longer
accepted even in “traditional” investment
literature (Bower, 1970; Kaplan, 1986). In
addition, the shortcomings of a purely financial
analysis are recognised in other specific segments
(Ansoff, 1971; Nagel, 1992). When examining
current investment methods, we notice that many
266
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
Question 3 Question 4
Clear Table Question 5
documentation? incomplete? Any comments?
Yes No N.a. Yes No N.a. Yes No N.a.
5 0 3 2 3 3 3 4 1
6 1 1 1 6 1 0 7 1
56 10 34 8 58 34 15 51 34
different data are presented, although the methods
differ strongly. It is our contention that the lack of
agreement is not the result of an insufficient
theoretical insight into factors influencing ICT
investment decisions. It is more likely the result
of the different circumstances in which
different organisations find themselves, which
dictate that other arguments have to be used
to underpin an investment decision.
A different version of this thesis is suggested by
Irani (2002) who suggests a number of criteria
based not on the (type of) organisation but on the
type of system involved, in his case MRPII
systems.
This in turn means that each organisation will
have to decide which data will be used in the
definition process. It is, however, possible to
provide a picture of the structure of the type of data
that can be considered. This structure can be used
as a starting-point when designing a local
approach.
Such an approach should at least account
for the financial consequences of the proposed
investment (costs and earnings). Explicit care
should be taken to define this data in such a way
that they match the current reporting
procedures and are in line with locally
accepted usage of language. Next, relevant
non-financial data should be selected and, last
but not least, relevant risk factors should be
included.
We feel that it was no coincidence that the
approach presented by Parker et al. need to
be adapted. Local circumstances will determine
what is relevant here. In order to support this
selection process a result of the literature survey
shown in Section 2 is presented in Table V,
which shows the types of non-financial criteria
encountered. This generic data set can form the
basis for a set of data that are specific to an
organisation.
A second lesson learned in this study deals
with process. The notion of viewing the
collection of data and evaluation aspects of defining
an ICT investment proposal as a human
information – processing activity paid off. Based on
this notion a number of process requirements were
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
Table V Overview of data
Category Examples
Costs Development
Training
Implementation
Operations
Maintenance
Security
Phasing out
Earnings Cost savings
Time savings
Increased output
Different output
Effectiveness Product/service quality enhancement
Increased customer service
Management information and decision
making
Improved communications
Competitive necessity
Competitive advantage
Increased market share
Transformation New markets and products/services
Influencing the value chain
Redesigning the business processes
Technology Technological necessity
Technological flexibility
Infrastructure availability
Integration and standardisation
Compliance Political pressure
Legal necessity
Wider human and organizational Organizational learning
impacts Political and cultural impacts
New work contents
Organizational structure
Risk area Examples of risk factors
Development Commitment of stakeholders
Application risk
Technological risk
Development staff
External suppliers
Implementation Resistance to change
Conversion
Lack of application knowledge
External Competitive risk
Structural economic risk
identified that were used as a starting-point for the
design of a process for collecting and evaluating
data (Table III). The evaluation discussed in the
previous section shows that this process is suited for
its purpose, which suggests that it is advisable to
observe these requirements. The usefulness of
these requirements was not tested only in the area of
cost-benefit analysis. Similar results were
obtained by experimenting in the fields of risk
management (Heemstra and Kusters, 1996) and
requirements engineering (Heemstra et al., 1995).
We feel that the lessons identified here can also be
267
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
used to strengthen current successful
approaches such as benefit analysis (Ward and
Peppard, 2002).
References
Argyris, C. (1983), Reasoning, Learning and Action:
Individual and Organization, Jossey-Bass,
San Francisco, CA.
Argyris, C. (1990), Overcoming Organizational Defenses:
Facilitating Organizational Learning, Allyn and Bacon,
Boston, MA.
Ansoff, H.I. (1971), Corporate Strategy, McGraw-Hill,
New York, NY.
Bedell, E.F. (1985), The Computer Solution: Strategies for
Success in the Information Age, Dow (Jones & Irwin),
Homewood, IL.
Bower, J.L. (1970), Managing the Resource Allocation Process,
Harvard University.
Brealy, R.A. and Myers, S.C. (1988), Principles of Corporate
Finance, McGraw-Hill, New York, NY.
Davis, G.B. (1982), “Strategies for information
requirements determination”, IBM Systems Journal,
Vol. 21 No. 1.
Frielink, A.B. (1961), Economics of Informatics, North-Holland,
Amsterdam.
Heemstra, F.J. and Kusters, R.J. (1996), “Dealing with risk: a
practical approach”, Journal of Information Technology,
Vol. 11, pp. 333-46.
Heemstra, F.J. and Kusters, R.J. (2000), “Assessing
IT-investments: costs, benefits, risks”, Project control:
The Human Factor, Proceedings of the 11th European
Software Control and Metrics Conference, 18-20 April,
Munich, Shaker Publishing, Maastricht.
Heemstra, F.J., Kusters, R.J. and Trienekens, J.J.M. (1995),
“Defining systems quality: involving end-users”,
in Rosca, I.G. and Ivan, I. (Eds), Software Engineering and
Applications, Proceedings of the 2nd International
Symposium of Economics Informatics, Ase, Boekarest,
pp. 110-21.
Irani, Z. (2002), “Information systems evaluation: navigating
through the problem domain”, Information and
Management, Vol. 20, pp. 11-24.
Janis, I., Kreitner, R., Angelo, K. and Buelens, M. (1999),
Organizational Behavior, 1st European Edition,
McGraw-Hill, Maidenhead.
Joslin, E.O. (1986), Computer Selection, Addison-Wesley,
Reading, MA.
Kaplan, R.S. (1986), “Must CIM be justified by faith alone?”,
Harvard Business Review.
McGrath, J.E. (1984), Groups: Interaction and Performance,
Prentice-Hall, Englewood Cliffs, NJ.
Mintzberg, H. (1983), Structures in Five: Designing Effective
Organizations, Prentice-Hall, Englewood Cliffs, NJ.
Nagel, A.P. (1992), “Increasing strategy innovation ability”,
(in Dutch), Thesis TUE.
Nonaka, I. and Takeuchi, H. (1995), The Knowledge Creating
Company, Oxford University Press, Oxford.
Parker, M., Benson, R. and Trainor, H. (1989),
Information Economics, Prentice-Hall, Englewood Cliffs,
NJ.
Renkema, T. and Berghout, E. (1997), “Methodologies for
information-systems evaluation at the proposal stage: a
comparative review”, Information and Software
Technology, Vol. 39 No. 1, pp. 1-13.
 Defining ICT proposals
Fred J. Heemstra and Rob J. Kusters
Renkema, T. (1999), The IT Value Quest: How to Capture the
Business Value of IT-based Infrastructure, Wiley, Chichester.
Strassmann, P.A. (1990), The Business Value of Computers,
The Information Economics Press, New Canaan, CT.
Ward, J. and Peppard, J. (2002), Strategic Planning for
Information Systems, Wiley, Chichester.
Weick, K.E. (1979), The Social Psychology of Organizing, 2nd ed.,
Addison-Wesley, Reading, MA.
Willcocks, L. (1994), Information Management: The Evaluation of
Information Systems Investments, Chapman & Hall,
London.
268
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 258–268
van Zanten, W.P.C. (1996), Group Decision in Management and
Administration (in Dutch), H. Nelissen/Open Universiteit,
Baarn/Heerlen.
Further reading
Nagel, A.P. (1983), “Increasing strategy innovation ability
(in Dutch)”, PhD thesis, Eindhoven University of
Technology.
 Introduction
A hedonic wage Usually, organisations in the public-sector are
model-based faced with challenges differing from those of
private firms. They have to meet multiple, often
methodology for conflicting goals such as providing better service
evaluating the benefits with reduced budgets and reduced staff, or they
are forced to provide a certain service to persons
of IT investments in entitled to it by law, without consideration of
economic aspects. Moreover, public-sector
public-sector organisations have a limited ability to use freed
capacity for introducing newly developed products
organisations or services. Several of these challenges hold true
for the private sector as well, but in the latter case
Peter Cilek an overriding goal of profit maximisation exists.
Moreover, public-sector organisations are subject
Wolfgang Janko to constraints of financial, legal/regulatory,
Stefan Koch contractual, personnel and institutional nature.
Normally, those constraints are much more
Andreas Mild and binding than in private firms (Kock and McQueen,
Alfred Taudes 1996; Luck and Peabody, 2000). For example, due
to legislative constraints, there might be no
possibility to reduce staff according to a new
The authors
situation, which indicates that any time saved
Peter Cilek, Wolfgang Janko, Stefan Koch, Andreas Mild due to the introduction of a new technology
and Alfred Taudes are all based at the Institute for Information has to be fully reallocated to other duties.
Processing and Information Economics, University of Economics Nevertheless, also in the public-sector the use of
and BA, Vienna, Austria.
information technology increases and, in fact, is
seen as a possibility to achieve higher efficiency.
Keywords
In this context, an evaluation model therefore,
Information systems, Public sector organizations, has to determine the most effective way of
Investment appraisal, Cost benefit analysis spending the time saved and the resulting value
for the organisation and not to calculate the
Abstract
savings by staff reductions. So far, the lack of
The economic justification of investments in information staff reduction following an information
technology (IT) is a basic issue for IT management in private and technology (IT) investment or a re-engineering
in public-sector organisations. Usually, the expenses made for attempt has sometimes been seen as a sign for
any investment are compared to the cost saved. While the costs
failure (Kock and McQueen, 1996). Using only
for implementing a new system are uncertain, only a small
the number of layoffs as a measure for evaluating
percentage of the benefits accrues as cost savings given the type
of IT systems used today and the particularities of the public- the success of an IT investment might not
sector. In this paper, we present a methodology for the monetary represent the whole picture, as any time saved
quantification of the benefits resulting from the introduction of a might be used more effectively, thus generating
modern IT application and demonstrate its use on the basis of additional value for the organisation. In the case
a case of prison administration. we describe in this paper, a public-sector
organisation is considered which faces the
restrictions mentioned above.
Electronic access
A taxonomy of evaluation techniques,
The Emerald Research Register for this journal is introduced by Bannister and Remenyi (1999),
available at will be used to constitute the choice of the
www.emeraldinsight.com/researchregister methodology to be applied. Three basic groups
The current issue and full text archive of this journal is of evaluation methods are distinguished.
available at (1) Fundamental methods attempt to parameterise
www.emeraldinsight.com/1741-0398.htm one single characteristic or a set of closely
related characteristics down to one single
measure or score. The most common scores
The Journal of Enterprise Information Management are financial measures.
Volume 17 · Number 4 · 2004 · pp. 269–275
q Emerald Group Publishing Limited · ISSN 1741-0398 The authors thank the co-editors and two anonymous
DOI 10.1108/17410390410548670 referees for their valuable comments and suggestions.
269
 A hedonic wage model-based methodology
Peter Cilek et al.
(2) Composite methods combine several
fundamental measures to get an overall
picture of an investment’s effects.
These methods include approaches like the
Balanced Scorecard or Portfolio Methods.
(3) Meta methods help to select a suitable
evaluation method for a certain decision,
depending on the structure of the problem.
An example for a meta method is described in
the work of Farbey and Targett (1994).
As most IT implementations also entail the
redesign of the respective business processes,
different process design scenarios need to be
compared. To this end, a quantitative evaluation of
the strategy and performance of the organisation’s
business processes is required (MacArthur et al.,
1994). Regarding quantitative measures,
monetary terms are to be preferred to other scores,
since only a monetary result of an evaluation
allows the application of investment analysis
methods. Of course, this is necessary to compare
an IT investment with other investment
possibilities (Hares and Royle, 1994; Hauser,
1996). Hence, no meta methods can be used in our
case, because they are only useful for the selection
of an appropriate evaluation model and do not
provide any ranking of the processes considered.
On the other hand, composite methods cannot be
used, since they will give only ordinal utility results
allowing to sort the actions of choice and not
a result on a cardinal scale. This indicates that we
have to use one or more fundamental methods
delivering a financial measure for the evaluation of
the new technology.
Of course, not all success factors of an
enterprise can directly be expressed in monetary
terms (Remenyi et al., 2000). Especially, this holds
true for public-sector organisations. They have to
provide services for which sometimes no kind of
market exists. As a consequence, non-monetary
benefits can be capitalised by estimating their
impact on the organisation’s market situation,
e.g. by calculating the raise of the contribution
margin caused by providing better service to
customers. Owing to this fact, we have to find
other micro- or macro-economic measures
allowing a transformation of qualitative benefits
to monetary terms.
One approach to quantifying changes in
efficiency and effectivity in administrative
processes is the hedonic wage model as introduced
by Sassone (1984, 1987). It is based on a
times-savings-times-salary model (TSTS) which
estimates the return of an IS as the time saved
multiplied by an average salary. In contrast to the
TSTS model, the hedonic wage model is able to
value returns caused by an increased effectiveness
as well. It has been applied several times in case
270
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 269–275
studies examining the benefits of office
automation systems and CAD systems (Janko et al.,
1989; Redlein and Rohrhofer, 1995;
Sassone, 1987; Stickel, 1992; Taudes and
Lukanowicz, 1994).
In the next section, we give a short introduction
to the hedonic wage model. Its application
necessitates a differentiation of salaries and work
profiles which is not always given, especially in
public-sector organisations where one type of
employee is responsible for all activities.
We therefore, propose a modification of this model
and demonstrate its usage via a case study on
IT implementation in prison management.
The hedonic wage model
Description
Sassone’s (1984, 1987) hedonic wage model is
based on the assumption that every employee
carries out a set of different types of activities, each
of them being of a different value for the
enterprise. The combination of activities
performed (and their respective share of total
working time) by an employee makes up his or
her “work profile”. An employee’s salary
(including fringe benefits, etc.) can be regarded
as the price paid for a certain bundle of activities
represented by the work profile. This salary is
known to the organisation.
In order to apply the hedonic wage model, three
assumptions about the economic model of the firm
have to be made:
(1) management allocates resources efficiently,
(2) workers allocate their time among activities
efficiently, and
(3) work comparable in value to current work
remains to be done (Sassone, 1987).
It is also assumed that for each category of
employees (e.g. managers, professionals and
secretaries) there is a corresponding set of activities
proper to that category. This means that there is
a set of activities that can be categorised as
management-type work, another one that can be
called professional-type work, and so on. In this
way, if there are N categories of employees, there
are N corresponding sets of activities plus one
additional activity called non-productive.
Non-productive activities, such as searching for
information, copying or travelling to and from
meetings are assumed to provide no positive
contribution to the company’s profit (Sassone,
1987).
The following notation will be used to
explain the theoretical basis of the method
(Sassone, 1987):
 A hedonic wage model-based methodology The Journal of Enterprise Information Management
Peter Cilek et al. Volume 17 · Number 4 · 2004 · 269–275

i ¼ 1,. . .,I ¼ index of categories of employees each category is supposed to be optimal under the
j ¼ 1,. . ., J ¼ index of sets of activities present technical and organisational conditions
Aj ¼ amount of the jth activity in the and, therefore, the budget constraints are not
firm (measured in man-years or in binding (Sassone, 1987).
another dimension) The first step for the practical application of the
MVj ¼ marginal value for the firm of hedonic wage model is to identify the categories of
activity Aj employees including their respective salaries and
R(Aj,. . .,Aj) ¼ revenue for the firm when activity main activities in the firm or department. Later,
amounts Aj and j¼1, . . ., J are the key activities identified have to be grouped into
accomplished adequate categories. As a next step, the work
Li ¼ amount of labour used by the profiles for the different categories of employees
organisation in the I category of have to be set-up. They can be achieved by logging
employees (measured in the same the time spent on various activities, questionnaires,
dimension as Aj) or the use of simulation models.
Pij ¼ work profile matrix, where Pij
gives the percentage of time spent Example
by workers Li doing activity Aj Janko et al. (1989) have described an example for
B ¼ budget for Li, i¼1, . . ., I applying the hedonic wage model in the context
u ¼ Lagrange multiplier of the introduction of an office automation
To maximize the firms profit, the following system in university setting. An analysis has shown
problem must be solved: that there are four categories of employees

X 
(scientific manager, scientific specialist, scientific
max RðA1 ; . . .; Aj Þ 2 C i · Li assistant and secretary) who perform four main
i
categories of work (managerial, senior specialist,
subject to junior specialist and administrative). Based on a
X
Aj ¼ P ij · Li for j ¼ 1; . . .j questionnaire, the work profiles shown in Table I
i have been derived (a difference of 100 per cent is
X the fifth category of non-productive work, which
B$ C i · Li is not shown):
i
For example, the scientific manager spends
The first assumption indicates that employees 39 per cent of his time on managerial duties like
allocate their time efficiently, whereas the preparing and making decisions, co-ordination
second constraint represents the budget etc., 36 per cent on specialist work, 10 per cent on
limitation. The relevant Lagrange expression can junior specialist tasks and 3 per cent on
be stated as administrative work. The remaining 12 per cent

X  X  are non-productive.
Z ¼R P i1 · Li ; . . .; P iJ · Li : As a next step, the (hedonic) price for one unit
i i
of each category of labour ( pman, psen, pjun and
X   X  padmin) can be calculated, using the work profiles
2 C i · Li 2 u· B 2 C i · Li and the employee category’s average loaded
i i salaries, by solving the resulting system of
equations:
The first-order conditions are
X dR 0:39pman þ 0:36psen þ 0:10pjun þ 0:03padmin
dZ
¼ · P ij 2 ð1 þ uÞ · C i ¼ 0
dLi i dAj þ 0:12pn2p ¼ 70; 000

0:10pman þ 0:40psen þ 0:26pjun þ 0:12padmin

for j ¼ 1; . . .; J
þ 0:12pn2p ¼ 50; 000
With dR=dAj ¼ MVj these conditions can be
rewritten as Table I
X Category of labour
P ij · MVj ¼ ð1 þ uÞ · Ci for i ¼ 1; . . .; I
Category of Mana. Sen. spec. Jun. spec. Admin.
i
employee (per cent) (per cent) (per cent) (per cent) Salary
By solving the resulting system of equations, the
marginal values for the different types of activities Sc. Manager 39 36 10 3 70,000
can be computed. In practice, the value for the Sc. Specialist 10 40 26 12 50,000
Lagrange multipliers u is often assumed to be zero. Sc. Assistant 2 20 43 23 35,000
Secretary 0 0 18 70 20,000
This means that the current number of workers in
271
 A hedonic wage model-based methodology
Peter Cilek et al.
0:02pman þ 0:20psen þ 0:43pjun þ 0:23padmin
þ 0:12pn2p ¼ 35; 000
0:00pman þ 0:00psen þ 0:18pjun þ 0:70padmin
þ 0:12pn2p ¼ 20; 000
0:00pman þ 0:00psen þ 0:00pjun þ 0:00padmin
þ 1:00pn2p ¼ 0
Solving the resulting set of equations gives the
hedonic values (Table II) for the different
categories of labour (the value for non-productive
work is zero):
Now, the revised work profiles have to be set-up,
since introducing a new technology will cause a
shift in the work profiles by replacing low-value
activities with duties in a category of higher value
(i.e. shifting the percentages of the working time).
This can be interpreted as the change in
effectiveness and measured in monetary terms by
means of the hedonic prices through calculating
the values of all new work profiles and summing
the differences to the old situation. A raise in
efficiency can be measured by calculating the
“equivalent hours”, which indicate the time taken
in the old process to do the same amount of work.
For example, an employee spending 30 h a week on
professional work with a new system that provides
a 10 per cent raise in efficiency would have needed
33 h to produce the same output when using an old
and inefficient system.
The projected new work profiles after
implementation of the office automation system is
assumed to be achieved over a period of 5 years by
linear improvement from the date of
implementation. This accounts for some effort
for learning the new system the first time after
introduction. Applying an interest rate of
8 per cent, the value of the system was estimated at
7,362,000 monetary units.
Limitations
In public-sector organisations, there are often no
different categories of employees but one type that
performs all types of activities. This violates the
assumption that there are different categories of
activities and employees within the organisation,
whereby each class of employees can be
characterised by a group of activities.
Table II
Category of labour
Mana. Sen. spec. Jun. spec.
Hedonic value 102,960 72,290 32,120
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 269–275
However, the hedonic values can only be
calculated if the number of categories of labour
equals the number of categories of employees
(in fact, the equations need to be linear
independent). Otherwise, the system of equations
has no unique solution, and the hedonic wage
model cannot be applied. Redlein and Rohrhofer
(1995) use the average hedonic values of the
industry concerned in the case of small
organisations where not all types of employees are
present. However, this is only possible if a market
price for the specific categories exists.
Description of the methodology
Assumptions
The following assumptions are made for the new
methodology proposed. Instead of assuming that
different categories of activities are carried out by
different classes of employees, we allow for all
activities to be performed by one single class of
employees with the same salary. Therefore, we
allow for situations where only one type of
employee exists, which will more often be the case
in public-sector organisations. Furthermore, we
assume that it is possible to compute a relationship
between the different categories of activities and
the achievement of goals of the organisation.
The main idea of the proposed methodology is
to calculate pseudo hedonic values using these
relationships by evaluating the “external” effects of
a shift in the working profile.
Model
The following steps are performed.
.
Defining goals of the organisation and
appropriate metrics for them (for example, the
goal/question/metric approach (Basili and
Rombach, 1988) developed for software
measurement or a similar framework could be
used to guide this process). For example, the
goals of prison management might include
reducing the offences committed while jailed.
This might be best represented by the number
of days of additional custody imposed due to
these offences.
.
Defining a set of n categories of activities
within the organisation.
.
Retrieving the current vector
a ¼ ða1 ; a2 ; . . .; an Þ of the amount of labour
per activity category measured in man-months
or any other appropriate measure.
The methods possibly used include
questionnaires, interviews, analyses of the
Admin. existing data like time sheets, multi-moment
observation studies, or other similar
20,310 techniques.
272
 A hedonic wage model-based methodology
Peter Cilek et al.
.
Computing the goal function of the
organisation from past data. The goal
function is a function r of the vector a.
Then the value of the current vector r(a)
can be computed. The coefficients of the
different activities are the pseudo hedonic
values.
.
Estimating the possible time saved after the
implementation of the information system.
This is best done after the implementation of
at least a working prototype, using direct
measurement of new activity times of the
employees. Other possibilities include
alternatives to direct measurement after
the implementation, e.g. questionnaires,
or working with the development team and
especially users who are part of it and who
might already give first estimates as to the
possible saving. Of course, their input might
be distorted due to their respective
motivations. Using the work profiles of similar
organisations as described by Redlein and
Rohrhofer (1995) might also be possible but
fails when the information system considered
is unique or similar organisations are missing.
This latter point might especially apply for
the public-sector, where a search for similar
organisations would need to span several
countries which might not be comparable due
to legal differences.
.
Defining any constraints in the new
distribution of working time. For example,
a fixed ratio between the professional activities
and secretarial work could be assumed.
.
Maximising the value of the new vector a0 by
maximising the function r(a0 ) subject to the
constraints defined above. In a simple case,
this results in a linear program.
.
Comparing the values of r(a) and r(a0 ).
The difference indicates the additional value
of the information system. Furthermore, the
new distribution a0 gives the optimal usage for
the time saved.
Many of these steps have been performed already
during a business process analysis as a basis for the
development of a new system. For example, many
data concerning the activities and maybe even
process times will already be available. Also, the
goals of the organisation and metrics for them
might be known from other activities like the
introduction of a Balanced Scorecard (Kaplan and
Norton, 1996). The data for constructing the goal
function will also be available, as historical data
are explicitly called for. In most projects, the
possible time saved will have to be determined,
which probably will be the cause for the biggest
effort in using the proposed methodology.
The other steps are mathematical computations,
273
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 269–275
which can easily be accomplished by a trained
individual.
Example
The following example details the application
of the proposed model within the environment of
prison management (due to privacy reasons, the
figures have been modified for this presentation).
In the country considered, prison management is
governed by federal law and performed by the
Ministry of Justice. The benefits of an investment
in an IT system covering several administrative
functions like reception, dismissal and payroll
administration for prisoners were to be evaluated.
Layoffs of employees were not considered as all
functions prescribed by law have not yet been fully
implemented due to a lack of staff. In addition,
layoffs would also have been very difficult, if not in
many cases impossible, due to legal constraints.
The following three major activities have been
defined: administration, custody and social care.
The vector of current distribution of labour has
been retrieved using a questionnaire sent to all
prisons. The time to be gained by the
implementation of the information system was
analysed based on the interviews with experts from
the field which were part of the software
development team and first results from a test
deployment of the software. The experts from the
field were prison employees from several prisons
who participated in the development as users and
testers, i.e. they belonged to the category
considered. Table III shows the distribution of
work before the possible introduction of the new
system and the expected new distribution (the new
values for administration and non-productive):
While it was clear that the system would reduce
the administrative effort and non-productive time
(mostly due to reduction in time waiting for
information), the optimal allocation of the
remaining time and, thus, the value of the
information system had to be calculated.
For the calculation of pseudo-hedonic values,
typical costs arising during detention had to be
identified. The costs resulting from drug abuse
(medical treatment) and costs caused by additional
custody imposed due to administrative offences
are considerably high. Therefore, the goal function
was to be constructed to minimise these costs by
optimally locating custody and social care. We have
also considered including any effects of the work
profile on relapse, but the costs associated with this
are not easily measured, as costs for the new jail
term do not include all aspects. Especially the state
faces high additional costs, as the offence removes
the perpetrator from the work force, thus resulting
in a loss of taxes and plus any costs associated with
the offence itself, in the worst case human life.
 A hedonic wage model-based methodology
Peter Cilek et al.
Table III
Administration
Old absolute values 500
Percentage 12.50
New absolute values 400
Then, the relationship between custody and social
care activities and the occurrence of these costs
had to be determined. Based on internal statistics,
we found a negative correlation between both
activities and resulting costs. Of course, one can
also use external studies in this process.
For example, several performance measures have
been developed for the criminal justice system,
(Logan, 1993). Especially for recidivism, several
prediction models have been developed (Maloof,
1999).
However, the allocation of time also has to take
into account several constraints. For example,
social care activities also require an additional
amount of custody, or a minimum level of custody,
has to be ensured. On the basis of this information,
which was either retrieved by interviews or derived
from internal directives, such constraints were
formulated.
The optimal allocation was then found by
solving the following program:
min K ¼ Ca ða* SC þ b* CUÞ þ Cb ðc* SC þ d* CUÞ
subject to the constraints:
CU $ CUmin
CU
$f
SC
where SC are the hours allocated to social care,
CU the hours for custody, Ca and Cb are the costs
for one drug abuse and one additional day of
custody imposed due to administrative offences,
respectively, a,b,c,d are the factors determining
the estimated occurrence of the additional costs
for a given number of hours for SC and CU.
Solving the program now gives the optimal new
allocation of hours (Table IV). Furthermore, the
difference in total costs can be discounted and
gives the value of the information system.
Conclusions and future directions
In this paper, we have presented a new
methodology for evaluating IT investments in
Table IV
New absolute 400 2,400 900
New percentage 10.00 60.00 22.50
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 269–275
Custody Social care Non-productive
2,600 500 400
65.00 12.50 10.00
– – 300
situations where other approaches cannot be
applied, especially in public-sector organisations.
The main restriction often faced in the
public-sector is that layoffs of employees are not
possible due to legal or political restraints.
Therefore, the gains in productivity made due to
the introduction of a new information system
cannot be quantified by calculating the salaries of
these employees.
As a result, the hedonic wage model has been
proposed to arrive at a quantification of the
resulting shifts in the employees’ work profiles
from activities of lower value to activities of higher
value. While this model is successful in doing so,
it cannot be applied if the number of employee
categories is smaller than that of the activities.
Especially in public-sector organisations, it is often
the case that one type of employee is responsible
for fulfilling all kinds of duties, thus preventing the
usage of this approach.
We have, therefore, presented a new
methodology related to the hedonic wage
approach. This methodology uses the main idea of
the hedonic wage model for evaluating the benefit
of an information system based on the changes in
the value of the employees’ work profiles using the
hedonic prices for each activity. To overcome
the limitations described, we calculate the hedonic
prices not based on a system of equations derived
from different employee categories, but from
“external” effects using past data. Most of the data
necessary are also a prerequisite for other methods
like the balanced scorecard or re-engineering
initiatives.
We have demonstrated the application of this
methodology in the context of prison
management, where the main effects of changing
a work profile, i.e. less administrative work and
more social care, were reduced costs for medical
treatment and less detention due to offences in
prison.
As each step in the evaluation process is
potentially associated with a certain level of
uncertainty, project controlling should carefully
analyse whether the parameter estimates proved
are reasonable in view of the business process data
observed. If this is not done, project advocates
might have an incentive to overstate certain project
benefits.
300 It would be interesting to see future research
7.50 applying the methodology presented to additional
274
 A hedonic wage model-based methodology
Peter Cilek et al.
public-sector organisations showing similar
restrictions. Conceivable examples could include
police work in public safety, where again one type
of employee is responsible for all types of activities
especially including administration, health care, or
organisational units performing standard clerical
work like issuing of passports.
Besides these efforts, future developments
for the approach taken with the presented
methodology can also be undertaken. For example,
the methodology could be adapted for evaluating
or even initiating changes in work profiles not
due to information system implementations.
In this way, the methodology could be used for
evaluating process innovations or even strategy
changes.
References
Bannister, F. and Remenyi, D. (2000), “Value perception in
IT investment decisions”, The Electronic Journal of
Information Systems Evaluation, Vol. 3, available at:
http://is.twi.tudelft.nl/ejise (accessed 26 May 2000)
Basili, V.R. and Rombach, H.D. (1988), “The TAME project:
towards improvement-oriented software environments”,
IEEE Transactions on Software Engineering, Vol. 14 No. 6,
pp. 758-73.
Farbey, B. and Targett, D. (1994), “Matching an IT project with
an appropriate method of evaluation: a research note on
evaluating investments in IT”, Journal of Information
Technology, Vol. 9, pp. 239-43.
Hares, J. and Royle, D. (1994), Measuring the Value of
Information Technology, Wiley, Chichester.
Hauser, C. (1996), Marktorientierte Bewertung von
Unternehmensprozessen, Verlag Josef Eul, Bergisch
Gladbach.
Janko, W., Pönighaus, R. and Taudes, A. (1989), “Der nutzen von
büroautomation – eine fallstudie”, Angewandte
Informatik, No. 10, pp. 436-45.
Kaplan, R.S. and Norton, D.P. (1996), The Balanced Scorecard:
Translating Strategy into Action, Harvard Business
School Press, Boston, MA.
275
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 269–275
Kock, N.F. Jr and McQueen, R.J. (1996), “Is re-engineering
possible in the public sector? A Brazilian case study”,
Business Change and Re-engineering, Vol. 3 No. 3,
pp. 3-12.
Logan, C.H. et al. (1993), “Criminal justice performance
measures for prisons”, in DiIulio, J.J. Jr, et al. (Ed),
Performance Measures for the Criminal Justice System:
Discussion Papers from the BJS-Princeton Project,
NCJ-143505, Bureau of Justice Statistics, Princeton
University, NJ.
Luck, J. and Peabody, J.W. (2000), “Improving the public sector:
can re-engineering identify how to boost efficiency at a VA
medical center?”, Health Care Management Review,
Vol. 25 No. 2, pp. 34-44.
MacArthur, P.J., Crosslin, R.L. and Warren, J.R. (1994), “A strategy
for evaluating alternative information system designs
for business process re-engineering”, International
Journal of Information Management, Vol. 14 No. 4,
pp. 237-51.
Maloof, M.A. (1999), “A machine learning researcher’s foray
into recidivism prediction”, Technical Report,
CS-99-02, Department of Computer Science,
Georgetown University.
Redlein, A. and Rohrhofer, R. (1995), “The use of CAD-systems –
economic and organizational impact”, in von Barsani,
K.R., MacConaill, P.A. and Thierney, K. (Eds), Opening
Productive Partnerships, IOS Press, Amsterdam.
Remenyi, D., Money, A., Sherwood-Smith, M. and Irani, Z.
(2000), The Effective Measurement and Management of
IT Costs and Benefits, 2nd ed., Butterworth-Heinemann,
Oxford, UK.
Sassone, P.G. (1984), “Cost-benefit analysis for office
information systems”, Proceedings of the International
Conference on Office Automation, pp. 145-53.
Sassone, P.G. (1987), “Cost-benefit methodology for office
systems”, ACM Transactions on Information Systems,
Vol. 5 No. 3, pp. 273-89.
Stickel, E. (1992), “Verfahren zur ermittlung der
wirtschaftlichkeit von informationssystemen”, in
Schweiggert, F. (Ed.), Wirtschaftlichkeit von Software-
Entwicklung und-Einsatz, Teubner, Stuttgart, p. 47.
Taudes, A. and Lukanowicz, M. (1994), “Cost-benefit analysis of
CAD Systems – a hedonic wage approach”, The 10th ISPE/
IFAC International Conference on CAD/CAM, Robotics and
Factories of the Future, CARs&FOF’94, 21-24 August 1994,
Ottawa, Canada.

Information systems
evaluation and the
information systems
development process
Paul Beynon-Davies
Ian Owens and
Michael D. Williams
The authors
Paul Beynon-Davies, Ian Owens, Michael D. Williams are all
based at the European Business Management School, University
of Wales Swansea, Singleton Park, Swansea, UK.
Keywords
Information systems, Modelling, Life cycle costs
Abstract
In this paper, we consider the synergy between two areas of
information system (IS) literature: that concerned with the
evaluation of IS and that concerned with explaining the
phenomenon of IS failure. On the basis of an analysis of both
areas, a model is presented which attempts to integrate IS
evaluation into the life cycle of IS development. The model links
the issue of failure assessment with the evaluation process and
constitutes a strategy for stimulating organisational learning in
relation to IS development. The paper concludes with a
description of our attempts to validate aspects of the model and
plans for further empirical work in this area.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 276–282
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548689
276
1. Introduction
According to Willcocks (1992), evaluation may be
defined as “establishing by quantitative and/or
qualitative means the worth of information
technology (IT) to the organisation”. The notion
of worth is inherently associated with definitions of
success or failure. In terms of information systems
(IS) we can distinguish between the worth of the
product (the IS), and the worth of the process
(the activities involved in producing the IS).
In practice, it is clearly difficult to separate the two.
The worth of the IS development process is
normally evaluated in terms of an assessment of
some features of the worth of the product.
Evaluation is critical to process organizational
learning in terms of IS/IT in that it is only through
effective evaluation that an organisation may
develop an effective knowledge based on which
successful development practice can be achieved.
Current evidence suggests that the worth of IS
(both as a product and a process) is open to
question (Brynjolfson, 1993) that evaluation is not
treated seriously by organisations, and
consequently that organisational learning in terms
of IS/IT is limited. For instance, Willcocks (1992)
finds a fall-off in evaluation at the project stages
following feasibility, little attempt to link
evaluation across the life cycle of systems and a
fragmented approach to learning from the
evaluation experience. He recommends an
integrative approach to evaluation across the life
cycle. However, there is little in the way of a
coherent set of recommendations as to how this
integration is to be achieved.
The aims of the paper are as follows. First, to
review the literature on IS evaluation, particularly
in the light of lessons learned from the areas of IS
failure and organizational learning. Secondly, to
present a preliminary model that attempts to
integrate approaches to IS evaluation into the life
cycle of the development process, incorporate the
ideas of failure assessment into the evaluation
process, and build a number of organizational
feedback loops into the model with the aim of
promoting organizational learning. Finally,
current experience and plans for validating
elements of this model in further empirical work
are discussed.
2. Existing literature
In this section, we give a brief review of lessons
learned from two areas of IS literature: that
concerned with the evaluation of IS and that
concerned with explaining the phenomenon of
IS failure.
Information systems evaluation and the information systems development
Paul Beynon-Davies, Ian Owens and Michael D. Williams
2.1 Lessons learned from the literature on IS
evaluation
According to Willcocks (1996), contemporary IS
literature appears to be focused upon issues of
evaluating the utility of an IS (that is to say, does
the information deliver key business benefit for the
organisation?). The literature tends to assume the
evaluation of two other dimensions against which
an IS may be evaluated, namely functionality (does
the IS do what is required?) and usability (is the IS
useable by its intended population?). Assessing the
degree to which a system is functionally complete
and consistent is a classic concern of the systems
development community. Assessing the usability
of systems has become increasingly important with
the continuing progress and use of graphical user
interfaces and multimedia interfaces.
Assessing the utility of IS is something which
most organisations conduct at the pre-
implementation stage of a project, but seldom
subsequently. Pre-implementation evaluation is
becoming increasingly important due to greater
pressure being placed on the IS function to more
closely account in financial terms for its activities.
Hence, both practice of IS evaluation and research
conducted in this area focuses primarily on
strategic evaluation (Ballantine and Stray, 1998).
The extant literature provides little practical
advice on how evaluation should be integrated into
the life cycle of IS development. This literature
appears to offer a less advice, for instance, as to
how proposed practices may be integrated with
traditional issues of project management. We
believe that the effective integration of evaluation
with life cycle issues is likely to lead to a greater
adoption of evaluation approaches amongst
organisations.
Although rarely explicitly stated as such,
evaluation of both IS and the processes by which
they are developed are inherently associated with
the issues of IS success and failure (Lyytinen and
Hirschheim, 1987). However, there appears to be
a lack of material which attempts to integrate
lessons from the literature on the issue of IS failure
with lessons from the material on IS evaluation.
The limited evidence that exists suggests that
organisations appear to engage little with the issue
of IS evaluation, and appear to learn little from IS
successes and failures (Owens and Davies, 1999).
What learning does occur appears to be in the form
that Argyris and Schon (1978) refer to as single-
loop learning. In single-loop learning, individuals
respond to error by modifying strategies and
assumptions within constant organizational
norms. Such learning is directed at increasing
organizational effectiveness. In double-loop
learning, response to detected error takes the form
of a joint inquiry into the organizational norms
277
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 276–282
themselves, the purpose being to resolve the
inconsistency between the existing norms and
make a new set of norms realisable (Figure 1).
There currently appears to be a less evidence of
organisations engaging in double-loop learning as
far as IS development practice is concerned.
2.2 Lessons learned from the literature on IS
failure
IS failure and success (DeLone and McLean,
1995) are clearly opposite sides of the same coin.
Having insufficient space available to discuss the
material on IS success here, we concentrate on the
issue of IS failure.
A survey conducted by the US Government’s
Accounting Agency (1985) in 1979 (ACM) found
that less than 3 per cent of the software that the US
government had paid for was actually used as
delivered. More than half of the software was never
used at all. In an international survey conducted by
Coopers and Lybrand (1996), 60 per cent of
organisations internationally and 67 per cent of
organisations within the UK had suffered at least
one systems project that had failed to deliver
planned business benefits or had experienced cost
and time overruns.
It is important to understand that failure is not
an objective concept. The definition of failure
depends on the position and perspective of the
definer – it is an intersubjective concept. Hence,
Lyytinen and Hirschheim’s (1987) concept of
expectation failure is critical. Expectation failure
refers to the inability of an IS to meet a specific
stakeholder group’s expectations.
As a consequence of the definition of
expectation failure, the identification of
stakeholders and their likely impact on the
trajectory of an IS project is extremely important.
Figure 1 Single and double loop learning
Information systems evaluation and the information systems development
Paul Beynon-Davies, Ian Owens and Michael D. Williams
IS failures signify a gap between some existing
situation and a desired situation for members of a
particular stakeholder group. Stakeholders are any
group of people who share a pool of values that
define what the desirable features of an IS are, and
how they should be obtained.
The trajectory of a project is defined as the
historical shaping of an IS both before and after
delivery. Frequently, the shape of an IS is
determined by the power play between different
stakeholder groups (Ballantine and Stray, 1998;
Keen, 1981). It should be recognised that an IS,
and an IS project, are a significant power-resource
in organisations. Failure can occur prior to the
delivery of the IS. This is the notion of project
abandonment or what Lyytinen and Hirschheim
(1987) refer to as development failure. Sauer’s
(1993) conception of termination failure
corresponds to the idea of total abandonment of a
project. However, projects may be substantially or
partially abandoned. In this case, the goals of the
IS may be reduced or reconfigured. Failure may
also occur after an IS has been delivered to its user
community. This is the idea of use failure
(Lyytinen, 1988). Use failure normally occurs
because the end-user stakeholders feel that the
IS does not match their expectations (Kling and
Iacono, 1984).
IS projects are frequently the subject of
escalation in decision-making. Drummond (1994)
defines escalation as “the predicament where
decision-makers find themselves trapped in a
losing course of action as a result of previous
decisions. Costs are incurred; there is an
opportunity to withdraw or persist; and the
consequences of withdrawal or persistence are
uncertain. Typically, the response to such
dilemmas is irrational persistence”. The important
point about escalation is that support for an
IS project can continue even in the face of major
system flaws. Major stakeholders in a project may
be reluctant to withdraw support due to heavy
investment in personnel and other resources
devoted to a project (Keil, 1995; Newman and
Sabherwal, 1996).
3. A model of IS evaluation
A model of IS evaluation closely aligned to the life
cycle of IS development is shown in Figure 2. This
model is based on, but much extended from that
proposed by Ewusi-Mensah and Przansnyski
(1995). It constitutes a preliminary attempt to
utilise the lessons from both literature on IS failure
and IS evaluation and attempts to specify
necessary organizational processes which embody
the importance of evaluation to processes of
278
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 276–282
organizational learning and the avoidance of IS
failure.
3.1 Importance of the model
This model is important for a number of reasons.
It incorporates processes of evaluation more
closely into the development activities of
organisations. It does this by distinguishing
between a number of distinct types of evaluation
and defining their rightful place within the
development life cycle. The model builds upon the
lessons learned from the extensive literature on IS
failure, particularly the fact that the participation
of representatives of stakeholder groups (including
end-users) is critical to all processes of evaluation.
Stakeholders need to be the explicit evaluators of
IS products. It emphasises a number of feedback
loops that encourage elements of organizational
learning and it acknowledges the fact that
development failure is in a commonplace. Failure
is not just about total abandonment of projects.
It is important for organisations to evaluate
reasons for the substantial or partial abandonment
of development projects.
3.2 Types of evaluation
The model makes a distinction between four types
of IS evaluation activity, primarily in terms of when
they take place in a standard linear model of the IS
life cycle. First, strategic evaluation, which is
sometimes referred to as pre-implementation
evaluation. This type of evaluation primarily
involves assessing or appraising an IS/IT
investment in terms of its potential for delivering
benefits against estimated costs. To do this some
preliminary idea of the functionality and likely
usability of the system needs to be established.
Second, formative evaluation involves assessing
the shape of an IS whilst in the development
process itself. Formative evaluation may be used to
make crucial changes to the design of an IS or to
make critical decisions concerning the degree of
project abandonment. Although formative
evaluation will primarily review issues of
functionality, this assessment will continuously be
shaped by notions of a systems usability and utility
formulated in the changing context of some
organisation.
The third type of evaluation identified occurs
after an IS has been implemented. For this reason
it is sometimes referred to as post-implementation
evaluation. Traditional approaches to summative
evaluation involve signing off some system against
its specification. More recently, there has emerged
an emphasis on usability testing. Ideally,
summative evaluation involves returning to the
costs and benefits established in strategic
evaluation after a period of use of the IS.
Information systems evaluation and the information systems development
Paul Beynon-Davies, Ian Owens and Michael D. Williams
Figure 2 IS evaluation and organisational learning
Summative evaluation may also produce ideas for
new systems and/or components. Finally, we
identify post mortem analysis. This is a variant of
summative evaluation particularly directed at
organisations learning from their partially or
totally abandoned projects.
4. Discussion
The model (Figure 2) is an attempt to build clear
formulations of distinct forms of evaluation
appropriate to various parts of the IS development
life cycle. In particular we distinguish between the
processes of strategic, formative and summative
evaluations. Strategic evaluation is a necessary pre-
development activity, formative evaluation is a
necessary part of the development activity itself
and summative evaluation a necessary post-
development activity. We also include in the model
a necessary evaluation activity which, following
Ewusi-Mensah, we have chosen to call here, post-
mortem analysis, which is a variant of summative
evaluation. This type of evaluation is essential in
developing an understanding of the reasons for full
or partial abandonment of IS projects.
4.1 Strategic evaluation
Most organisations conduct some form of strategic
evaluation of IS projects (Ballantine and Stray,
1998). Strategic evaluation may be conducted as
part of strategic planning process, more usually as
part of some project selection or feasibility study.
Strategic evaluation is designed primarily to
279
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 276–282
evaluate aspects of utility and cannot be done
without some early notions of functionality.
Strategic evaluation is an activity that attempts
to establish the balance of predicted costs and
benefits in terms of an intended IS project. We have
called it strategic evaluation because of its ideal
role in determining elements of both long-term
and short-term IS strategy. Strategic evaluation is
normally used to initiate a go/no-go decision in
terms of a given development project. It may also
be used to prioritise a number of IS investments.
The most popular techniques applied in this
process are return on investment and payback period.
Such techniques are effective ways of evaluating
tangible costs against tangible benefits. One of the
most popular strategic evaluation frameworks that
include facilities for an assessment of intangible
costs and benefits is Information Economics
(Parker and Benson, 1988).
4.2 Formative evaluation
Formative evaluation should be an inherent part of
the project management process in the sense that
development and evaluation should be parallel
activities. Development projects should be
continually assessed against objectives and careful
attention should be paid to this activity to avoid
project escalation. Project escalation is defined as
the continued commitment to an IS project in the
face of continual negative information from
formative evaluation exercises. Major stakeholders
in an IS project may be reluctant to withdraw
support because of heavy investment in personnel
and other resources devoted to a project.
Information systems evaluation and the information systems development
Paul Beynon-Davies, Ian Owens and Michael D. Williams
Formative evaluation may focus on issues of
process or that of the product. Traditional
approaches to project management tend to
emphasise the formative evaluation of process
elements. The emphasis is in decomposing a
project into a series of tasks or activities that are
evaluated against performance at regular intervals.
Alternative approaches to formative evaluation,
popular in such development methods such as
rapid applications development (RAD), take a
product-based focus. Here, the emphasis is in
defining a series of project deliverables and
negotiating the so-called time-boxes (fixed
deadlines) for the delivery of these system products
(Stapleton, 1997).
4.3 Post-mortem analysis
At some point in the development process a
decision may be made to either wholly or partially
abandon a project (what Lyytinen and Hirschheim
call development failure).
In terms of development failure (project
abandonment), Ewusi-Mensah and Przasnyski
(1995) distinguish between three types of project
abandonment. Complete termination of all
activities on a project prior to full implementation
is referred to as total abandonment. Substantial
abandonment describes major truncation or
simplification of the project to make it radically
different from the original specification prior to full
implementation. Finally, partial abandonment is
used to describe reduction of the original scope of
the project without entailing significant changes to
the IS original specification, prior to full
implementation.
In each case, the organisation should engage in
another form of IS evaluation called here a post-
mortem analysis. Collier and DeMarco (1996)
equates the idea of what they call a post-mortem
review with summative evaluation. They quote one
of Alan Davies’ 201 principles for software
development:
Principle 172: Conduct a post-mortem. . .. At the
end of every project, give all the key project players
a three or four-day assignment to analyse every
problem that occurred during the project.
This should attempt to determine the key reasons
for such total or partial failure. The results of such
analysis are important in suggesting ways in which
the organisation may improve its development
practice. It is for this reason that the document
produced from such a post-mortem analysis needs
to be disseminated to senior management, project
management and members of the project team.
This, of course, can only be done effectively if
assurances of non-recrimination are given to all
project participants. Ideally, a reputable senior
executive that was not involved in the project
280
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 276–282
under consideration should conduct this analysis.
Alternatively, an external body or consultant
should undertake it.
Wherever possible post-mortem information
should be made public. This is important in
enabling the validation of IS development practice
and the effective progression of the profession of IS.
4.4 Summative evaluation
The figure also emphasises another important
organizational learning feedback loop. Even if a
project reaches completion, it may fail in some
sense when it comes to be delivered (what
Lyytinen calls use failure).
Therefore, at some suitable time after a system
has been delivered the organisation should engage
in a summative evaluation of the system and its
project. One framework proposed for the
summative evaluation of IS is benefits management
(Ward, 1990). Even at this point it is possible that
the system may be wholly or partially abandoned,
in which case it should also be the subject of a post-
mortem analysis, as above.
It is important to emphasise that no system is
ever complete. A summative evaluation is likely to
suggest a number of ways in which the system may
be modified or extended – normally both classed
as systems maintenance. This is the third feedback
loop illustrated on the diagram. The conclusion is
that effective evaluation leads to effective
management of maintenance.
Kumar (1990) conducted an empirical study of
the prevalence and form of evaluation of IS after
they have been implemented amongst major
companies in the US. Three major results are
evident from the data he collected. The major
reason for performing post-implementation
evaluation amongst the companies he surveyed
was the formalisation of the completion of the
development project. Evaluation was treated as a
major tactic in a project disengagement strategy.
Much of the evaluation was managed and
performed by those who had designed the system
being implemented. The most frequently
evaluated criteria seem to be that of information
quality criteria (accuracy, timelines, adequacy and
appropriateness) along with facilitating criteria
such as user satisfaction and attitudes. Socio-
technical criteria such as the system’s impact on
the user and the organisation were evaluated much
less frequently.
Hirschheim and Smithson (1988) maintain that
although there appears to be widespread
agreement regarding the need to evaluate both
products and processes of IS work, there is little
agreement as to the appropriate ways of
conducting such evaluations. In one sense, the area
of evaluation is a very wide one, encompassing
Information systems evaluation and the information systems development
Paul Beynon-Davies, Ian Owens and Michael D. Williams
many processes that take place during the life of an
IS project, including selection, procurement and
testing. In their analysis of the literature on post-
implementation evaluation they organise the
material on a dimension beginning at those which
take a highly rational and objective stance on the
evaluation process and ending at those which
regard evaluation as very subjective or political.
They identified three approaches to evaluation.
Those who approach evaluation in terms of
efficiency assume that the function and goals of
evaluation are non-controversial and that the
overall aim is to achieve precise measurement of
performance, efficiency or reliability. A second
approach is to look at evaluation in terms of
effectiveness. This approach attempts to evaluate
the effectiveness of IS in terms of usage or utility,
costs and benefits, or some notions of increases in
job satisfaction. The third approach looks at
evaluation in terms of understanding. Here, the
aim is to understand the function and aims of
evaluation itself, particularly how evaluations are
performed within the political and social
environment of an organisation.
Evaluation in terms of understanding is clearly
the perspective that lies closest to our emphasis on
stakeholders and their expectations. A project
should be continuously monitored in terms of the
degree to which it is likely to satisfy stakeholder
expectations. Also, at the completion of a project,
the project needs to be evaluated in terms of what
it tells the organisation about development
processes.
Ideally, the issue of evaluation should be built
into a project profile from the start. One clear way
of doing this is to build in clear measures of success
at the start of the project. This should indicate at
some appropriate point clear and straightforward
measurements we make to assess whether we have
been successful or not.
4.5 Empirical study of evaluation activity in
real-world settings
Compared to a vast amount of prescriptive
material published there is little empirical work on
IS evaluation (Ward and Taylor, 1996). An early
attempt to validate elements of the model is
described in some detail in Owens and Davies
(1999). In this paper, an historical analysis of the
evaluation activity conducted within a large-scale
development project was undertaken. The paper
examined the development and implementation
of a two-phase mission-critical IS that spanned
a decade of development work conducted by
OTIS PLC.
The conclusions of this study found that a
strategic evaluation had been undertaken as part of
the initiation activity for phase one of the
281
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 276–282
development project, but there was little evidence
of systematic formative evaluation activity being
conducted during this phase and no systematic
summative evaluation was conducted after the
implementation of phase one. An informal
assessment of the success of the IS was made at the
time, primarily in terms of meeting its functional
specification and some vague expressions of user
satisfaction with the system.
A number of problems gradually emerged in the
2 years of use of the IS after phase one and a
second phase of development work was initiated.
One could argue that this was an example of single-
loop learning. However, no strategic evaluation is
evident for this second phase. No form of
summative evaluation was conducted after the
implementation of the second phase of the project.
This study provides some evidence that
organisations fail to take IS evaluation seriously.
It is important to determine why this is the case.
Some of the reasons applicable to OTIS were
found to be a lack of sufficient people with the
requisite expertise able to conduct effective
evaluation; pressures exerted on the IS service by
the business to get the system in quickly; and the
perceived low status of evaluation activity amongst
both business and IT staff within OTIS.
5. Conclusions and further work
In this paper, we have considered some of the
inherent relationships between the literature on
IS evaluation: that on IS failure and that on
organizational learning. From such an analysis we
have developed a model of IS evaluation that arises
from this work and distinguished between a
number of distinct types of IS evaluation. We have
demonstrated a number of anchorage points for
lessons from the literature on IS failure and
indicated how each type of evaluation fits with the
IS development life cycle.
Part of the reason for the low adoption rates of
evaluation activity within organisations may be the
current gap between other established IS
development (ISD) approaches, methods,
techniques and tools and the suggestions produced
by the IS evaluation community. One direction for
further work is therefore, to consider the degree to
which it is feasible to integrate evaluation activity
with existing methods for IS planning, project
management and IS development. More specific
questions and areas for further work are described
below.
To what extent can we integrate our IS
evaluation model with conventional ISD methods
like structured systems analysis and design method
(SSADM) or unified modelling language (UML)?
To attempt this we first aim to convert our model
Information systems evaluation and the information systems development
Paul Beynon-Davies, Ian Owens and Michael D. Williams
into a more generic and transportable form as
discussed in Dori (1995). We would like to
discover as what degree do project management
methods currently acknowledge evaluation as a
process and to what extent can evaluation be
melded with such approaches?
We would also like to discover to what extent
does our model of evaluation fit with different
models of the development life cycle than the
linear model utilised in this paper. It might be
argued that that the separation of evaluation from
development activity is encouraged by linear
models of the life cycle. However, our interest is in
particularly examining large-scale development
projects that while being the most likely to fail
(Cash et al., 1992) are also the most likely to be
conducted in a linear manner. However, it is
particularly important to assess the degree of
efficacy of applying the model in iterative
approaches such as rapid application development
(Stapleton, 1997), particularly where such
approaches are scaled-up to address larger
projects.
A key question we wish to pursue in further
work is to what degree organisations conform to
our ideal-type? For actual projects, can we track
the history of strategic, formative and summative
evaluation activities in organisations? We are
currently employing the model presented in this
paper to evaluate a large IS development project at
the National Assembly for Wales.
References
Argyris, C. and Schon, D. (1978), Organisational Learning: A
Theory of Action Perspective, Addison-Wesley, Reading,
MA.
Ballantine, J.A. and Stray, S. (1998), “A comparative analysis of
the evaluation of information systems and other capital
investments: empirical evidence”, European Conference
on Information Systems, Aix-en-Provence.
Brynjolfson, E. (1993), “The productivity paradox of information
technology”, Comm. ACM, Vol. 36 No. 12, pp. 67-77.
Cash, J.I., Mcfarlan, F.W. and McKeney, J.L. (1992), Corporate
Information Systems Management, Richard D. Irwin,
Homewood, IL.
Collier, B. and DeMarco, T. (1996), “A defined process for project
post-mortem review”, IEEE Software, pp. 65-72.
Coopers and Lybrand (1996), Managing Information and
Systems Risks: Results of an International Survey of Large
Organisations, Technical Report.
DeLone, W.H. and McLean, E.R. (1995), “Information systems
success: the quest for the dependent variable”,
Information Systems Research, Vol. 3 No. 1, pp. 60-95.
Dori, D. (1995), “Object-process analysis: maintaining the
balance between system structure and behaviour”,
Journal of Logic and Computation, Vol. 5 No. 2, pp. 227-49.
Drummond, H. (1994), “Escalation in organisational decision-
making: a case of recruiting an incompetent employee”,
Journal of Behavioural Decision-Making, Vol. 7, pp. 43-55.
282
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 276–282
Ewusi-Mensah, K. and Przansnyski, Z.H. (1995), “Learning from
abandoned information system development projects”,
Journal of Information Technology, Vol. 10, pp. 3-14.
Hirschheim, R. and Smithson, S. (1988), “A critical analysis of
information systems evaluation”, in Bjorn-Anderson, N.
and Davis, G.B. (Eds), Information Systems Assessment:
Issues and Challenges, North Holland, Amsterdam,
pp. 17-37.
Keil, M. (1995), “Pulling the plug: software project management
and the problem of project escalation”, MIS Quarterly,
Vol. 19 No. 4.
Keen, P. (1981), “Information systems and organisational
change”, Comm. ACM, Vol. 24 No. 1.
Kling, R. and Iacono, S. (1984), “The control of IS developments
after implementation”, Comm. ACM, Vol. 27 No. 12,
pp. 1218-26.
Kumar, K. (1990), “Post implementation evaluation of computer-
based information systems: current practices”, CACM,
Vol. 33 No. 2, pp. 236-52.
Lyytinen, K. and Hirschheim, R. (1987), “Information systems
failures: a survey and classification of the empirical
literature”, Oxford Surveys in Information Technology,
Vol. 4, pp. 257-309.
Lyytinen, K. (1988), “The expectation failure concept and
systems analysts view of information systems failures:
results of an exploratory study”, Information and
Management, Vol. 14, pp. 45-55.
Newman, M. and Sabherwal, R. (1996), “Determinants of
commitment to information systems development: a
longitudinal investigation”, MIS Quarterly, pp. 23-54.
Owens, I. and Davies, P.B. (1999), “The post-implementation
evaluation of mission-critical information systems”,
European Conference of Information Systems,
Copenhagen.
Parker, M. and Benson, R. (1988), Information Economics:
Linking Business Performance to Information Technology,
Prentice-Hall, NJ.
Stapleton, J. (1997), DSDM – Dynamic Systems Development
Method: The Method in Practice, Addison-Wesley, Harlow,
England.
Sauer, C. (1993), Why Information Systems Fail: A Case Study
Approach, Henley-On-Thames, Alfred Waller.
US Government Accounting Agency (1985), “US Government
Accounting Office Report, FGMSD-80-4”, ACM Sigsoft
Software Engineering Notes, Vol. 10 No. 5.
Ward, J.M. (1990), “A portfolio approach to evaluating
information systems investments and setting priorities”,
Journal of Information Technology, pp. 122-31.
Ward, J. and Taylor, P. (1996), “Evaluation and realisation of IS/IT
benefits: an empirical study of current practice”, European
Journal of Information Systems, Vol. 4 No. 1, pp. 214-25.
Willcocks, L. (1992), “Evaluating information technology
investments: research findings and reappraisal”,
Information Systems Journal, Vol. 1 No. 2.
Willcocks, L. (Ed.) (1996), Investing in Information Systems:
Evaluation and Management, Chapman & Hall, London.
Further reading
Ewusi-Mensah, K. (1991), “On information systems project
abandonment: an exploratory study of organisational
practices”, MIS Quarterly, Vol. 15 No. 1, pp. 67-85.
Hirschheim, R. and Newman, M. (1988), “Information systems
and user resistance: theory and practice”, Computer
Journal, Vol. 31 No. 5, pp. 398-408.

Narratives in ERP
systems evaluation
Jonas Hedman and
Andreas Borell
The authors
Jonas Hedman is based at the Department of Informatics,
School of Economics and Management, Lund University,
Sweden.
Andreas Borell is at Tetra Pak Global Information Management,
Sweden.
Keywords
Narratives, Information systems,
Manufacturing resource planning
Abstract
The purpose of evaluating enterprise resource planning (ERP)
systems ought to be to improve the systems’ life cycles. In this
paper, narratives are proposed as a means of improving ERP
systems as a complement to traditional evaluation methods.
The potential of narratives is that they can convey meanings,
interpretations, and knowledge of the system, which may lead to
action. Even though narratives belong to an interpretive research
tradition, this paper takes a pragmatic view of evaluation on the
basis of three assumptions about evaluation: evaluations should
form the basis for action; narratives can make evaluation more
relevant; and evaluations should be made with the purpose of
improving selection, implementation and use of the system. The
conclusion reached is that narratives can advance evaluation
practice by providing a richer evaluation picture which conveys
meanings not included in traditional evaluations, and improve
the use of ERP systems by changing users’ mental maps.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 283–290
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548698
283
Introduction
The purpose of evaluating enterprise resource
planning (ERP) systems, or any other form or type
of information systems (IS), ought to be to improve
their selection, development, implementation and
usage. In this paper, narratives are proposed as a
means of improving ERP systems, i.e. the systems’
life cycles. This paper explores the potential of
narratives and proposes their use in ERP systems
evaluation. The contribution of narratives is that
they can convey meanings, interpretations, and
knowledge (learning) of the system, which can be
used for further action. Interest in ERP systems
evaluation stems from the new problems and
challenges they are posing as regards to IS
evaluation (Stefanou, 2001). ERP systems affect all
aspects of a business, including organizational
structure, activities and processes, and the
environment (Davenport, 2000; Hedman and
Kalling, 2002; Hong and Kim, 2002).
Independently of ERP systems, IS evaluation is
a problematic (Irani and Love, 2001) and complex
process (Jones and Hughes, 2001), which becomes
more and more difficult with the increased
complexity of the IS (Farbey et al., 1995).
Furthermore, there is growing concern amongst
academics and practitioners that IS do not deliver
business value (Irani and Love, 2001; Kennerley
and Neely, 2001; Murphy and Simon, 2002a;
Poston and Grabski, 2001). Thus, the overall
research question is: “Can narratives be used as
means of improving ERP systems evaluation?”
This paper builds on research gleaned from
interpretive research in IS, e.g. Klein and Meyer
(1999) and Walsham (1993, 1995, 1999), on post-
modern institutional theory (Meyer and Rowan,
1977), and on narratives in accounting (Llewellyn,
1998), knowledge management (Snowden, 2002;
Swap et al., 2001), and requirements engineering
(Jark et al., 1998). The interpretive research and
post-modern institutional theory are primarily used
as theoretical grounds for the use and relevance of
narratives, whereas the latter is used as a source of
practical and methodological inspiration.
The approach to evaluate ERP implementation
in this paper is pragmatic: once a particular system
has been selected, the focus should be on
continuously improving the benefits received. This
pragmatic approach is based on three assumptions
about evaluation in this context. First, evaluations
should form the basis for action: do not measure if
you cannot act upon the measurement. Secondly,
evaluations should in practice be carried out with
the goal of improving selection, development,
implementation, and use of the system. Thirdly,
narratives can grasp the complexity of ERP
systems better than traditional evaluation
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
approaches, e.g. return on investment. Thus, the
aim of this paper is not to try to improve the
understanding of the evaluation process as such,
which is common in interpretive research
(Walsham, 1993), but to provide input regarding
how we can conceptually advance IS evaluation in
general, and ERP system evaluation in particular
(March and Smith, 1995).
The paper is organised as follows. The next
section addresses and discusses ERP systems,
evaluation, and the role of measurement.
Following this, the theory of action and learning is
described, which is the theoretical ground for
improvement-driven evaluation. Narratives and
the use of narratives in evaluations, which should
be interpreted as the means for action, are then
explored and discussed. The paper ends with a
discussion and conclusion regarding the practical
implications for ERP systems evaluation.
Evaluation and measurements
ERP systems and ERP systems evaluation
ERP systems have had, and will continue to have,
an enormous impact on businesses around the
world (Davenport, 2000). For instance, Nestlé
invested US $500 million on an ERP system.
Initially, the American subsidiary started on the
project in 1997, then spent over US $200 million
and will complete by 2003. In 2000, the global
parent decided to invest an additional US $200
million, plus US $80 million in consulting and
maintenance for a global solution (Worthen, 2002).
According to Nestlé’s CIO, the savings made by the
system have so far reached US $325 million for the
American subsidiary. This type of global
investment will affect the entire organisation – its
culture and structure, business processes,
strategies, and financial position (Davenport,
1998). The economic rationale for this type of
investment is that organisations believe that there is
more value in buying ERP systems than other IS.
There are other reasons for selecting a particular
system, including institutional pressure and
individual aspirations (Nilsson, 1991).
While it is reasonably easy to evaluate tangible
implementation costs, e.g. software licences,
hardware, consultancy, and training, other costs
like productivity dip and resistance to change are
much more difficult to measure (Murphy and
Simon, 2002b). It is uncertain whether ERP
systems contribute to any improvement. For
instance, Kennerley and Neely’s (2001) 2-year
study of an SAP R/3 implementation at four plants
in Europe by a multinational corporation showed
operational benefits by users, but the corporation
was still doubtful whether the system had
284
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
contributed to any significant positive financial
benefits.
Existing ERP systems research has mainly
addressed issues such as implementation (Markus
and Tanis, 2000; Ross and Vitale, 2000), critical
success factors (Hong and Kim, 2002; Nah et al.,
2001), business impact (Francalanci, 2001; Hitt
et al., 2002; Kennerley and Neely, 2001; Poston
and Grabski, 2001), implementation failures
(Scott and Vessey, 2000), and requirements
specification (Borell and Hedman, 2001; Rolland
and Prakash, 2000). Despite the impact and
organizational importance of ERP systems, ERP
systems evaluation is a neglected research area
(Stefanou, 2001). The are a few exceptions,
namely; Stefanou (2001) who proposes an ex ante
evaluation framework for ERP systems evaluation,
Murphy and Simon (2002a), who examine an
attempt to incorporate intangibles into traditional
cost-benefit analysis in an ERP project, and Borell
and Hedman (2001) who suggest the use of
interpretative evaluation approaches.
General IS evaluation: current state and
issues
Remenyi and Sherwood-Smith (1999) described
two practical issues in IS evaluation. The first is
known as the evaluation gap. This occurs when the
evaluators distance themselves from the project
and lose sight of the business objectives. For
instance, when the implementation project is
evaluated rather than the business impact of the
IS, the focus is on the means, not the ends.
The second issue is that the business objectives of
the IS project, e.g. organizational change and
change management, are often forgotten or
considered only superficially during evaluations. In
addition, evaluations may have many purposes,
e.g. control projects, governing change
management, communication, improvements,
resource allocation, motivation, and long-term
planning (Sinclair and Zairi, 1995). Another issue
in evaluations is when to evaluate – ex ante or ex
post. This dichotomy is less important today, since
there is an increased awareness of the importance
of an ongoing evaluation process (Remenyi, 1999).
The academic view of the purpose of IS evaluation
is to assess and improve the design and use of IS
(Remenyi and Sherwood-Smith, 1999). However,
Kumar (1990) found that, in practice, the purpose
of IS evaluation was often to close the project, with
the emphasis on ROI. The focus on project closure
and ROI has been confirmed by Seddon et al.’s
(2002) study on evaluation practice in Europe and
North America. Hirschheim and Smithson (1988)
argue that this is partly because of the widespread
belief that IS are fundamentally technical systems.
Consequently, this has led to “a more ‘technical’
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
interpretation of evaluation” (Hirschheim and
Smithson, 1998, p. 402) focusing on tools and
techniques and thereby omitting the social
domain. This makes it unlikely to produce a “true”
or meaningful evaluation picture (Hirschheim and
Smithson, 1998).
The art of measurements and measuring
We argue that the deeper meaning of evaluation is
“to do better next time” – i.e. evaluation has to be
followed by action. At the same time, evaluation
requires the use of measurements (Venkatraman
and Ramanujam, 1986), but “figures never are
facts” (Hoebeke, 1990). So, while Strassmann
(1985) stresses that: “You cannot measure what is
not defined. You also cannot tell whether you have
improved something if you have not measured its
performance”, it is important to remember that
numbers do not always tell the whole truth. Thus,
there is a need for an operational definition of a
measurement. But, as measurements are
invariably used in socially complex processes
wherein translations as well as interpretations take
place, the concept of relevance, as Hoebeke (1990)
points out, is a lot more important than objectivity.
In addition, there has to be a shared understanding
of the intentions, and their impacts, by those who
play a role in the process of collecting, translating
and interpreting the measurements, as well as
those who act on decisions based on the said
measurements. These activities are iterated, as can
be seen in Figure 1.
There have been many attempts to stratify or
classify IS measurements and evaluations. For
instance, Farbey et al. (1995) proposed a model
that related specifically to evaluation based on the
idea of a fit between organizational change and the
IS. The model consists of eight levels; mandatory
changes, automation, direct value systems, MIS
and DSS, infrastructure, inter-organizational
systems, strategic systems, and business
transformation. The classification is not rigid, but
still implies that, while higher levels of change
increase the potential benefits, they also increase
the uncertainty of the outcome. Potential benefits
and levels of uncertainty are cumulative. Systems
classified on a certain level may have all the
accumulated benefits and uncertainty from any or
all of the previous levels. They conclude that, for
Figure 1 Chain of activities taking place in organizational sense-making
285
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
the implementation of systems on the 8th level
(business transformation), “. . .benefits stem from
the transformation as a whole. IS provides only one
component of what is often a complex series of
changes. It is not possible to attribute a portion of
the benefits gained to any one factor” (Farbey et al.,
1995, p. 49).
We argue that, in the taxonomy of Farbey et al.
(1995) ERP systems have the potential impact of
the 8th level and thus, the potential benefits and
accumulated uncertainty of all the strata below.
Therefore, it is highly unlikely that any two
implementations will have identical requirements
or consequences, even if they are based on the same
generic software packages. While the potential
benefits might be articulated, it makes the actual
benefits of implementing an ERP system hard to
define, predict, and measure. Furthermore, ERP
system implementations are only limited in time in
the most abstract sense and governed by other
organizational activities – project plans, budget
years, top management changes and organizational
strategic decisions. Implementing an ERP system is
deeply interconnected with organizational change,
and, for the lifetime of the system, it remains a large
part of the organisation’s formalised internal
framework. Evaluation in this context should be
performed continuously, combined with process
reviews and organizational development which
aims to obtain the desired alignment and fit over
time. To get the most out of a large organisation, the
results of evolution have to be evaluated and new
requirements are formulated in an iterative process.
The key to getting the most out of any IS lies in its
use; use in this perspective is regarded in the
broadest sense, including other use, misuse, abuse
and non-use of the system by humans and
connected IS. To achieve the intended use of the
system in the organizational context, organizational
goals and beliefs have to be communicated to the
members of the organisation.
Theory of action and learning
However, human actions are not always what they
seem to be. Argyris and Schön (1974) suggest that
people act in accordance with their mental maps
rather than the theories they espouse. People are not
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
aware of the mental maps or theories they use
(Argyris, 1980). One could say that there is a split
between theory and action. Argyris and Schön
(1974) suggest that two theories of action are
involved. “When someone is asked how he would
behave under certain circumstances, the answer he
usually gives is his espoused theory of action for that
situation. This is the theory of action to which he
gives allegiance, and which, on request, he
communicates to others. However, the theory that
actually governs action is theory-in-use” (Argyris
and Schön, 1974, p. 6-7, our italics). As humans
invariably think and express themselves using the
full depth of language – by telling “stories” – it is
therefore conceivable that, unless external
observations “verify” the stories, the stories told
will not reflect theory-in-use but espoused theory.
Argyris (1980) makes the case that effectiveness
results from developing congruence between
theory-in-use and espoused theory. Reflection is a key
tool for revealing the theory-in-use and exploring
the gulf between espoused theory and theory-in-use,
or in bringing the latter to the surface. Provided
the two remain connected, the gap will create a
dynamic for reflection and dialogue (Figure 2).
A model of the processes involved is required in
order to fully appreciate theory-in-use. Argyris and
Schön (1974) initially looked at three elements:
governing variables, action strategies and
consequences. Argyris (1976) proposed the double
loop learning theory, which concerns the changing
of underlying values and assumptions, i.e.
learning, (Figure 3). The focus of the theory is on
solving problems which are complex and poorly
structured and which change as problem-solving
advances. Typically, interaction with others is
necessary to identify the conflict.
There are four basic steps in the action theory
learning process:
Figure 2 Espoused theory and theory-in-use
Figure 3 Double loop and single loop learning
286
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
(1) the discovery of espoused theory and theory-
in-use;
(2) the invention of new meanings;
(3) the production of new actions; and
(4) the generalization of results.
Double loop learning involves applying each of
these steps to itself. In double loop learning,
assumptions underlying current views are
questioned and hypotheses regarding behaviour are
tested publicly. The result of double loop learning
should be the increased effectiveness of action and
the improved acceptance of failures and mistakes.
Narratives
Narratives are used to persuade, convince, and
make people act and behave in certain ways. The
alternative to narratives for reasoning, learning,
and persuading is to use numbers and calculations,
including financial statements, investment
calculations and time reports (Llewellyn, 1998).
Clausen (1994, p. 45) states that: “Using
narratives in the system development process
seems to be a way in which designers will be able to
come up with the kind of descriptions that are
asked for.” Besides everyday life narratives that are
used by all people, a theoretical ground for
narratives can be found in institutional theory.
Meyer and Rowan (1977) suggest that
“rationalised myths” contribute to the
understanding of organisations. Narratives or
stories conveying myths are powerful tools that
make the irrational rational (Llewellyn, 1998).
Narratives are common in accounting,
management, human computer interaction (HCI),
knowledge management, strategic management,
and software engineering. Llewellyn (1998)
discusses how narratives are best understood,
constructed, and used in accounting and
management research. In HCI, research narratives
are used to improve communication between
end-users and developers in the design of user
interfaces, task modelling and prototyping, and in
supporting the specification of user interfaces
(Bødker, 2000). The role of narratives as the
conveyers of tacit knowledge is explored in
knowledge management (Swap et al., 2001).
In strategic management, scenarios are used to
explore future alternatives wherein scenarios are
“tools” in the “strategist’s arsenal” (Porter, 1985,
p. 481). Software engineering, on the other hand,
uses narratives and scenarios to gather and validate
requirements (Antón and Potts, 1998; Jarke et al.,
1998). It should be noted that there are different
underlying philosophical reasons for using
narratives. Bødker (2000), Clausen (1994) and
Llewellyn (1998) represent an interpretive
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
research tradition, whereas Antón and Potts
(1998), Jarke et al. (1998) and Porter (1985) works
can be classified as belonging to a more positivistic
research tradition.
Narratives have also emerged as an alternative
approach within IS research. Hirschheim and
Newman (1991) use the concept of myth to
interpret social processes during IS projects.
Clausen (1994) develops a model for how IS
designers can use narratives to make descriptions
of IS that people understand; traditional methods
such as structured languages and formal
specifications. Brown (1998) examines the use of
narratives to explain and create meaning in power
struggles within IS implementations. Dubé and
Robey (1999) analyse the stories of competing
groups in IS development projects as symbols of
organizational values or myths to gain insights into
the interpretation of management styles. Alvarez
and Urla (2002) describe the use of narratives in
ERP system requirements specification. Finally,
Alvarez (2002) examines the role of myths in
constructing an ERP system as an integrated
system and in elaborating the existing
organizational values.
Narratives regarding ERP implementation
projects can be written before, during and after a
project. Participants in the project, or external
auditors, can write narratives, making it clear that
one crucial component of using narratives in
evaluation is the means of keeping track of meta-
information about narratives, including their
perceived interdependencies.
Narratives written prior to a project would
typically be used to express requirements, as well
as to build and control expectation and support.
Narratives from this phase form the basis for
decisions and action, and assume the role of
defining the strategic objectives, vision and
mission of a project. During an implementation
project, narratives are used to express the project
status, the need for further improvements, and
refinement of scope, as well as to track the
decisions made. During the end phase of an ERP
implementation project (if there is such a thing),
narratives are used to persuade and justify; the
outcome of the project and the “defining of the
next steps” are often expressed in narratives, which
should then form the basis for future actions and
learning. Before, during and after a project, there is
thus a need to record, analyse, classify and, not
least, create narratives that are used to describe the
perceptions of a project’s current and future state.
Narratives can carry a lot of information and,
when handled with care, they can be used to
achieve procedural results and process
improvements at the same time. Used for
evaluation purposes, they are perhaps the
287
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
best-suited measurement vehicle for making
continuous improvements to both subject and
evaluation process. In fact, narratives might be
the only way to capture invaluable input regarding
a project of the size and complexity of an ERP
implementation.
Discussion
This paper suggests the use of narratives in ERP
systems evaluation by building on research gleaned
from interpretive IS research, post-modern
institutional theory, and the use of narratives in
reference disciplines. Narratives belong to the
interpretative research traditions and are used to
reason, teach, and persuade (Llewellyn, 1998).
Organisations use myths and narratives to make
sense of equivocal situations. They are also used as
precedents for future actions, serving as
“blueprints” for desirable behaviour (Alvarez and
Urla, 2002). The following section discusses the
practical implications of the proposed use of
narratives in ERP systems evaluation and some
research issues.
The actual implications for practice might not
be so great, since narratives are used in business to
convey meanings and to persuade people to act.
The use of narratives as an approach to measuring
and evaluating ERP systems might involve
paradigm shifts in the existing control system, as
well as how to evaluate IS. Changes to the existing
norms, behaviours, and procedures are difficult
(Weick, 1996). Resistance to change is likely to
emerge. This is a rational behaviour for those who
are affected by change, not a dysfunctional one
(Markus, 1983). The formalisation of narratives
might also create new positions, e.g. that of chief
storyteller. The storyteller might become the new
power position. Narratives are powerful tales that
can be manipulated by different stakeholders,
e.g. managers, project leaders, and storytellers.
The process of changing the organizational
culture so that it accepts narratives will be a
challenging process for most organizations and
business managers. Besides changing the culture,
there is also a need to develop support tools,
e.g. storyboards, and procedures to create, store,
and disseminate stories (Snowden, 2002; Swap
et al., 2001). There is a need for a lot of stories
along the system’s life cycle, with different
purposes. Based on Markus and Tanis’s (2000)
“Enterprise Systems Experience Cycle”, it is
possible to outline four types of stories during
the life cycle.
(1) Chartering stories are aimed at selecting and
funding an enterprise system. These stories
can convey the myths of the system, e.g. the
myths of integration (Alvarez, 2002), which
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
can be used to promote a certain system or the
opposite. Stories about the benefits of ERP
systems can promote the idea. Furthermore,
stories can be used to obtain the commitment
of top management.
(2) Project stories are aimed at supporting the
configuration and implementation of the
system. During this phase, the initial training
takes place. Thus, it is important to obtain the
acceptance of and commitment to the system
within the organisation to decrease user
resistance and transfer knowledge. Swap et al.
(2001) described the role of mentors (super
users) and stories in knowledge transfer.
Furthermore, these stories should have two
target groups. One group is the consultants,
who actually carry out the development and
implementation. The other group is the users,
who require different types of stories. User
stories should include and convey the meaning
and role of the system by addressing questions
such as what problems we are solving, why we
selected this solution, how the system works
and what role the user plays in this.
(3) Shakedown stories are crucial, since these are
at the beginning of use and often associated
with productivity dips, which can be overcome
through extensive training. The aim of these
stories is to overcome problems and issues
associated with learning and using a new
technology.
(4) Onward and upward stories are aimed at
obtaining payback by improving the use. These
stories will continue to grow and be
disseminated until the enterprise system has
been replaced by a different system. Stories are
probably even used in the replacement of ERP
systems, e.g. all the Y2K horror stories used in
the replacement of legacy systems by ERP
systems. One example of a story used during
the shakedown and onward and upward phases
is provided by Austin et al. (2001):
Elizabeth Benson. . . turned quickly to her desktop
computer. She had just received word of fire in a
manufacturing plant belonging to a key vendor. . ..
Fast action was needed. . .. Three hours after she
had. . . processed information from Procurement,
Engineering, Inventory Management, Finance,
and Manufacturing. Without the high levels of
integration and standardization offered by Tristen’s
new information systems, days or weeks would
have passed before they could act. . .
This story might inspire other users of ERP
systems, within the organisation or within other
organisations, to investigate the possibilities of
ERP systems and ultimately change their mental
models, possibly leading to the improved use of
ERP systems. Narrative is a communication
medium with a high degree of media richness,
288
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
which can represent a suitable approach when
there is a high degree of uncertainty and
equivocality (Daft and Lengel, 1986; Daft et al.,
1987). Other stories might reveal institutionalised
work practices, system inefficiencies, and how
users manage these inefficiencies, and they might
also be used to convince consultants or managers
as regards to necessary changes to the system,
where narratives can be used as tools for making
the irrational rational.
Conclusion
A number of researchers have proposed
interpretive approaches to IS evaluation (Borell
and Hedman, 2001; Hirschheim and Smithson,
1988; Walsham, 1993). Hirschheim and Smithson
(1988) suggest that interpretative approaches are a
way of gaining a deeper understanding of the
process itself. Symons (1991) supports this by
suggesting that evaluation means understanding
the different perspectives of individuals, while
Walsham (1993, p. 179) states “interpretive
evaluation designs focus on learning and
understanding.” However, no one has yet explicitly
mentioned narratives as an interpretive evaluation
approach. Neither has “doing better next time”
(which would be a typical process improvement)
been replaced by “doing better all the time”.
Continuously learning about the possibilities and
difficulties is, we believe, the key to successfully
understanding an ERP before and during
installation, and its use after installation.
Narratives appear to be well suited as tools for
furthering the understanding of, as well as the
support for, system implementation projects.
Nevertheless, there are research implications
and opportunities in acknowledging narratives as
an evaluation approach to ERP and IS. The
implication is that narratives should not only be
used as an input into research, which is the
traditional input in case research, they should also
be used as a research output. Quantitative data can
also form the bases of narratives. For instance,
instead of stating the correlation between x and y
in statistical terms, this can be expressed in words.
The consequence of using narratives as a research
output is that researchers have to be able to
interpret and evaluate stories in the same way as
any other research output.
Thus, we conclude that ERP systems evaluation
is highly contextual, and that measurements of the
impact are interchangeable with the measurements
of general improvements. For such measurements
to be relevant, they will have to be used over a long
period of time, and measurements would have to be
initiated before the implementation is started
(Hoebeke, 1990), e.g. during the requirements
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
specification phase. Returning to the quote by
Strassmann (1985), we argue that we need to
measure the impact of ERP systems using the same
measurements employed in a specific organisation
prior to implementation, and that any changes
recorded will be attributable to a combination of IS
implementation, organizational changes and
changes in the method of measurement. Thus, the
measuring or quantifying of isolated impacts
by ERP systems implementation is close to
impossible. Instead of trying to measure the
impossible, we propose the use of narratives as
the means of disseminating knowledge, which
ought to lead to improvements and enhanced
action during the implementation and usage of
enterprise systems.
References
Alvarez, R. (2002), “The myth of integration: a case study of an
ERP implementation”, in Hossain, L., Patrick, J. and
Rashid, M. (Eds), Enterprise Resource Planning: Global
Opportunities and Challenges, Idea Group Publishing,
Hershey, PA, pp. 17-42.
Alvarez, R. and Urla, J. (2002), “Tell me a good story: using
narrative analysis to examine information requirements
interviews during an ERP implementation”, The DATA
BASE for Advances in Information Systems, Vol. 33 No. 1,
pp. 38-52.
Antón, A. and Potts, C. (1998), “A representational framework
for scenarios of systems use”, Requirements Engineering,
Vol. 3 Nos 3/4, pp. 219-41.
Argyris, C. (1976), Increasing Leadership Effectiveness, Wiley,
New York, NY.
Argyris, C. (1980), Inner Contradictions of Rigorous Research,
Academic Press, New York, NY.
Argyris, C. and Schön, D. (1974), Theory in Practice: Increased
Professional Effectiveness, Jossey-Bass, San Francisco, CA.
Austin, R.D., Cotteleer, M.J. and Escalle, C.X. (2001), Enterprise
Resource Planning: Technology Note, Harvard College,
Boston, MA.
Bødker, S. (2000), “Scenarios in user-centered design: setting the
stage for reflection and action”, Interaction with
Computers, Vol. 13 No. 1, pp. 61-76.
Borell, A. and Hedman, J. (2001), Artefact Evaluation of ES
Impact on Organizational Effectiveness, Americas
Conference on Information Systems, Boston, MA.
Brown, A. (1998), “Narrative, politics and legitimacy in an IT
implementation”, Journal of Management Studies, Vol. 35
No. 1, pp. 35-59.
Clausen, H. (1994), “Designing computer systems from a human
perspective: the use of narratives”, Scandinavian Journal
of Information Systems, Vol. 6 No. 2, pp. 37-44.
Daft, R.L. and Lengel, R.H. (1986), “Organizational information
requirements, media richness and structural design”,
Management Science, Vol. 32 No. 5, pp. 554-71.
Daft, R.L., Lengel, R.H. and Trevino, L.K. (1987), “Message
equivocality, media selection, and manager performance:
implications for information systems”, MIS Quarterly,
Vol. 11 No. 3, pp. 355-66.
Davenport, T.H. (1998), “Putting the enterprise into the
enterprise system”, Harvard Business Review, Vol. 76
No. 4, pp. 121-31.
289
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
Davenport, T.H. (2000), Mission Critical: Realizing the Promise of
Enterprise Systems, Harvard Business School Press,
Boston, MA.
Dubé, L. and Robey, D. (1999), “Software stories: three cultural
perspectives on the organizational practices of software
development”, Accounting, Management and Information
Technologies, Vol. 9 No. 4.
Farbey, B., Land, F.F. and Targett, D. (1995), “A taxonomy of
information systems applications: the benefits of
evaluation ladder”, European Journal of Information
Systems, Vol. 4 No. 1, pp. 41-50.
Francalanci, C. (2001), “Predicting the implementation effort of
ERP projects: empirical evidence on SAP/R3”, Journal of
Information Technology, Vol. 16 No. 1, pp. 33-48.
Hedman, J. and Kalling, T. (2002), IT and Business Models:
Concepts and Theories, Liber and Abstrakt, Malmö,
Sweden.
Hirschheim, R.A. and Newman, M. (1991), “Symbolism and
information systems development: myth, metaphor and
magic”, Information Systems Research, Vol. 2 No. 1,
pp. 29-62.
Hirschheim, R.A. and Smithson, S. (1988), “A critical analysis of
information systems evaluation”, in Bjørn-Andersen, N.
and Davis, G.B. (Eds), Information Systems Assessment:
Issues and Challenges, pp. 17-37.
Hirschheim, R.A. and Smithson, S. (1998), “Evaluation of
information systems: a critical assessment”, in Willcocks,
L. and Lester, S. (Eds), Beyond the IT Productivity Paradox,
pp. 381-409.
Hitt, L.M., Wu, D.J. and Zhou, X. (2002), “Investment in
enterprise resource planning: business impact and
productivity measures”, Journal of Management
Information Systems, Vol. 19 No. 1, pp. 71-98.
Hoebeke, L. (1990), “Measuring in organisations”, Journal of
Applied Systems Analysis, Vol. 17, pp. 115-22.
Hong, K.K. and Kim, Y.G. (2002), “The critical success factors for
ERP implementation: an organizational fit perspective”,
Information and Management, Vol. 40 No. 1, pp. 25-40.
Irani, Z. and Love, P.E.D. (2001), “Information systems
evaluation: past, present and future”, European Journal of
Information Systems, Vol. 10, pp. 183-8.
Jarke, M., Bui, X. and Caroll, J. (1998), “Scenario management:
an interdisciplinary approach”, Requirements Engineering,
Vol. 3 Nos 3/4, pp. 154-73.
Jones, S. and Hughes, J. (2001), “Understanding IS evaluation as
a complex social process: a case study of a UK local
authority”, European Journal of Information Systems,
Vol. 10 No. 4, p. 189.
Kennerley, M. and Neely, A. (2001), “Enterprise resource
planning: analysing the impact”, Integrated
Manufacturing Systems, Vol. 12 No. 2, pp. 103-13.
Klein, H.K. and Meyer, M. (1999), “A set of principles for
conducting and evaluating interpretive field studies in
information systems”, MIS Quarterly, Vol. 23 No. 1,
pp. 67-94.
Kumar, K. (1990), “Post-implementation evaluation of computer
based information systems (CBIS): current practices”,
Communications of the ACM, Vol. 33 No. 2, pp. 203-12.
Llewellyn, S. (1998), “Narratives in accounting and management
research”, Accounting, Auditing and Accountability
Journal, Vol. 12 No. 2, pp. 220-36.
March, S. and Smith, G.F. (1995), “Design and natural science
research on information technology”, Decision Support
Systems, Vol. 15 No. 4, pp. 251-66.
Markus, M.L. (1983), “Power, politics, and MIS
implementation”, Communications of the ACM, Vol. 26,
pp. 430-44.
 Narratives in ERP systems evaluation
Jonas Hedman and Andreas Borell
Markus, M.L. and Tanis, C. (2000), “The enterprise systems
experience-from adoption to success”, in Zmud, R.W.
(Ed.), Framing the Domains of IT Management: Projecting
the Future. . .. Through the Past, Pinnaflex Educational
Resources, Inc., Cincinnati, OH, pp. 173-207.
Meyer, J.W. and Rowan, B. (1977), “Institutionalized
organizations: formal structure as myth and ceremony”,
American Journal of Sociology, Vol. 82 No. 2, pp. 340-63.
Murphy, K.E. and Simon, S.J. (2002a), “Intangible benefits
valuation in ERP projects”, Information Systems Journal,
Vol. 12 No. 4, pp. 301-20.
Murphy, K.E. and Simon, S.J. (2002b), “Using cost benefit
analysis for enterprise resource planning project
evaluation: a case for including intangibles”, in Hossain,
L., Patrick, J.D. and Rashid, M.A. (Eds), Enterprise Resource
Planning: Global Opportunities and Challenges, Idea
Group Publishing, Hershey, PA, pp. 245-66.
Nah, F.F-H., Lau, J.L-S. and Kuang, J. (2001), “Critical factors for
successful implementation of enterprise systems”,
Business Process Management Journal, Vol. 7 No. 3,
pp. 285-96.
Nilsson, A.G. (1991), “Anskaffning av standardsystem för att
utveckla verksamheter: utveckling och prövning av SIV-
metoden”, Ekonomiska forskningsinstitutet vid
Handelshögskolan i Stockholm: Stockholm.
Porter, M.E. (1985), Competitive Advantage: Creating and
Sustaining Superior Performance, Free Press, New York,
NY.
Poston, R. and Grabski, S. (2001), “Financial impacts of
enterprise resource planning implementations”,
International Journal of Accounting Information Systems,
Vol. 2, pp. 271-94.
Remenyi, D. (1999), Stop IT Project Failure Through Risk
Management, Butterworth Heinemann, Boston, MA.
Remenyi, D. and Sherwood-Smith, M. (1999), “Maximise
information systems value by continuous participative
evaluation”, Logistics Information Management, Vol. 12
No. 1, pp. 14-31.
Rolland, C. and Prakash, N. (2000), “Bridging the gap between
organisational needs and ERP functionality”,
Requirements Engineering, Vol. 5 No. 3, pp. 180-93.
Ross, J.W. and Vitale, M.R. (2000), “The ERP revolution,
surviving vs thriving”, Information Systems Frontiers,
Vol. 2 No. 2, pp. 233-41.
290
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 283–290
Scott, J.E. and Vessey, I. (2000), “Implementing enterprise
resource planning systems: the role of learning from
failure”, Information Systems Frontiers, Vol. 2 No. 2,
pp. 213-32.
Seddon, P.B., Graeser, V. and Willcocks, L.P. (2002), “Measuring
organizational IS effectiveness: an overview and update of
senior management perspectives”, The DATA BASE for
Advances in Information Systems, Vol. 33 No. 2, pp. 11-27.
Sinclair, D. and Zairi, M. (1995), “Performance measurement as
an obstacle to TQM”, The TQM Magazine, Vol. 7 No. 2,
pp. 42-5.
Snowden, D. (2002), “Complex acts of knowing: paradox and
descriptive self-awareness”, Journal of Knowledge
Management, Vol. 6 No. 2.
Stefanou, C.J. (2001), “A framework for the ex-ante evaluation
of ERP software”, European Journal of Information
Systems, Vol. 10 No. 4, pp. 204-15.
Strassmann, P.A. (1985), Information Payoff: The Transformation
of Work in the Electronic Age, Free Press, New York, NY.
Swap, W., Leomard, D., Shields, M. and Abrams, L. (2001),
“Using mentoring and storytelling to transfer knowledge
in the workplace”, Journal of Management Information
Systems, Vol. 18 No. 1, pp. 95-114.
Symons, V.J. (1991), “A review of information systems
evaluation: content, context and process”, European
Journal of Information Systems, Vol. 1 No. 3, pp. 205-12.
Venkatraman, N. and Ramanujam, V. (1986), “Measurement of
business performance in strategy research: a comparison
of approaches”, Academy of Management Review, Vol. 11
No. 4, pp. 801-14.
Walsham, G. (1993), Interpreting Information Systems in
Organisations, Wiley, Chichester.
Walsham, G. (1995), “The emergence of interpretivism in IS
research”, Information Systems Research, Vol. 6 No. 4,
pp. 376-94.
Walsham, G. (1999), “Interpretive evaluation design for
information systems”, in Willcocks, L. and Lester, S. (Eds),
Beyond the IT Productivity Paradox, Wiley, Chichester.
Weick, K.E. (1996), “Drop your tools: an allegory for
organizational studies”, Administrative Science Quarterly,
Vol. 41 No. 2, pp. 301-31.
Worthen, B. (2002), “Nestlé’s ERP Odyssey”, CIO, available at:
www.cio.com/research/erp/

Other papers
Online point-of-click
Web usability mining
with PopEval (C-Assure
Co-design methodology)
Atta Badii
The author
Atta Badii is based at Intelligent Media Systems and Services
Research Group, Informatics Research Centre, University of
Reading, Massachusetts, USA.
Keywords
Online operation, User interfaces, User satisfaction
Abstract
Earlier research has shown that human judgements, particularly
retrospective as distinct from real-time evaluations of painful
experiences, are fundamentally flawed when superseded by
other experiences. Describes a methodology for the evaluation of
user experiences, assisted by a user-centered co-design tool for
planning, creating and conducting wide-ranging usability data
acquisition throughout the system life cycle from inception to
replacement. The results from the tests in this study were found
to support the findings from recent cognitive psychology
research on J/DM-PPR theoretic biasing effects. The importance
of the findings for software systems, and for interactive Web site
development, is that if the interactive features which tax the
user’s patience and cognitive capability are not severely
restraining, then users will be left with a better impression of the
usability of the system than would be the case otherwise.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 291–300
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548706
291
1. Introduction
Why should the use of information systems (IS)
and technology appear to so many as a challenging
and stressful part of life? Interactive software and
systems should be a lot easier for users to use and
less problematic for organisations to integrate into
their transformed processes for competitive
advantage. Figure 1 is a model of emerging
applications of interactive online systems and
services. It illustrates how for global optimisation
of online services management, as distinct from
shifting bottlenecks, new standards and tools are
needed, e.g. using online evaluation and profiling
tools such as PopEval as deployed here for
dynamic usability evaluation. In any service
scenario we need to find out what makes users tick/
click. For this, we need to understand what, if
anything, could suppress the memory of some
aversive experiences and contribute to some
non-aversive ones becoming more dominant and
persistent in the user’s memory and thus, in the
collective memory of a user group. Saliency and
recency effects in human memory and differential
foregrounding influences in memory re-call have
been the subjects of research in cognitive
psychology (Ahlawat, 1999; Lange-Kuettner and
Martin, 1999; Mahrer and Miles, 1999). Thus,
various correlations have been observed and
theories established on the effect of sequential order
(recency-primacy), or stimulus novelty and
significance (saliency), in memory recall as the
delay between exposure and test increases.
Similarly, the research on the psychology of human
J/DM-PPR has investigated the effects of any
sequential order and/or duration sensitivities on
human real-time or retrospective judgements of
serial experiences, which caused a sequence of
aversive and/or non-aversive affects. Thus, the
central motivation in our work here has been to
devise a tool to facilitate the study of the root
causes of, and the routes to, the aggregated
perceptions of system usability by users, over time;
i.e. users’ current salient impression (SI) of
(dis)satisfaction, at any point in the usability
relationship timeline Tn.
2. Dynamic usability evaluation;
theoretical foundations
Most reasonable people prefer any experience in
life to be as painless, and as fruitful as possible, be
it using a piece of software or having a surgical
operation (Kahneman and Fredrickson, 1993a).
The primary objective of an interactive system is to
allow the user to achieve particular goals in the
application domain, i.e. the interactive system
 Online point-of-click Web usability with PopEval
Atta Badii
Figure 1 Online service management architecture
must be usable (Dix et al., 1993). Usability is
defined as follows.
(1) Classic usability. A system is usable if it
incorporates learnability, flexibilty and
robustness (Dix, 1993c).
(2) Dynamic Usability (Badii, 1999) added three
new criteria to the above classic definition of
usability, as follows.
.
High Mutual Man-Machine Intelligibility
and Scanability (M*IS).
.
Mutually Productive Balance of Power,
Privacy and Patterns-of-Relating (MP*oR)
between the man and machine personas, as
transactors
.
High Sustainability of both M*IS and
MP*oR i.e. SMP*.
This dynamic usability expands on the classic
definition to provide a view of instantaneous or
transient usability, which is necessary in dynamic
process-centred analysis. It aids formulation and
testing of spatio-temporal-causal models of usability,
for it is more expressive of the complex psycho-
socio-technical dynamics of situated usability.
Further, the SMP* criterion allows it to explain the
effects of such J/DM-PPR theoretic saliency-
recency, on instantaneous versus steady-state or
quiescent usability. Thus, this dynamic usability is
292
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
sufficiently, richly defined to cope with the
increasingly volatile click-happy user environment
that pertains to WMS, for example, in online
shopping. It is important for manufacturers, users,
online services, and, change/churn managers to
have a facility for developing a high perceived
usability or comfort factor (PCF) for interactive
systems. Several approaches to evaluation exist as
addressed by researchers (Draper et al., 1996).
Our primary focus here is online user-led
evaluations as distinct from following traditional
HCI guidelines on general usability do’s and
dont’s, divorced from the situated context. These
HCI heuristics, which have remained unchanged
for at least the past 25 years, assess the extent to
which a system incorporates classically expected
interaction features which are considered by HCI
experts to be user-friendly in general, e.g. Direct
Manipulation (Shneiderman, 1982), and WIMP
interface (Dix et al., 1993). Contextually sensitive
usability evaluation is, on the other hand, what
characterises our approach here. Ideally we would
expect both smart (i.e. re-adaptive) interactive
systems and smart organisations running smart
projects to remain contextually aware with some
capability for reflexive reasoning and re-learning.
Users can share some common attributes and
 Online point-of-click Web usability with PopEval
Atta Badii
expectations as well as have attitudinal, mindset
and cultural idiosyncrasies, which require
contextually sensitive usability evaluation on a
quantitative, qualitative and interpretivist basis.
A prerequisite for global mass usability
improvement must be to ensure the existence of
contextually richly defined data from a large
enough user base to help mass-personalisation of
new technology development and the attendant
change management involved in its successful
adoption. So, in this research, we have studied the
results from J/DM-PPR, as sub-fields of cognitive
science to see what judgement theoretic insights,
regarding human bias in evaluations are applicable
to the contextually sensitive usability evaluation of
interactive systems. As indicated earlier, human
decision-making is in fact subject to many biases
(Kahneman et al., 1982) but the influence of such
biases have to date remained largely unexplored
within the HCI community. Thus, so far human
J/DM-PPR biases have not been formally explored
in relation to the study of interactive software
systems evaluation. The study of pleasure and pain
recall (PPR) has also been researched from a J/DM
perspective. The three most relevant pieces of
research here have looked into the effect of real-
time evaluations of affective episodes compared
with their retrospective evaluations (Kahneman
et al., 1993a, b; Redelmeier et al., 1996). Also
explored are the effects of “duration neglect” in
retrospective evaluations of affective episodes.
Significant insights exist on how time influences
our thoughts about affective experiences (Coombs
and Avrunin, 1977; Hsee, 1990/91; Johnson and
Sherman, 1990). The basic finding of Kahneman’s
research, as reported in two experiments, was that
a person’s global evaluation of an affective
experience is subject to the phenomenon of
“duration neglect”. His two experiments involved
32 subjects each, who were asked to watch a
selection of film clips that varied in length,
intensity and aversiveness. The results showed that
retrospective evaluations made from memory
seemed to underestimate the duration of an
aversive experience – in this case, an unpleasant
video clip. This would seem irrational as most
reasonable people would prefer to experience
shorter aversive experiences rather than a longer
one, but the results argue that the length
sometimes does not matter.
The importance of this result is that human
memory may not keep a comprehensive time based
“film” of events, but instead “photographs” or
snapshots, which represent experiences by the
selected salient points. Patients’ memories of
painful medical treatments were investigated by
analysis of real-time and retrospective
evaluations of two minimally invasive procedures
293
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
(Kahneman et al., 1993b). The findings from this
piece of research were that a person’s memories of
painful episodes are in fact imperfect and
susceptible to bias. Similar effects can be found in
research concerning judgements about painful
episodes (Algom and Lubel, 1994v; Corli et al.,
1986; Varey and Kahneman, 1992). Here the
above studies attempted to establish whether some
peak-and-end evaluation bias as well as duration
neglect bias, occur in the memories of patients who
have been through a painful medical procedure.
This indeed proved to be the case; the patients’
memories of a medical procedure were affected by
the most painful part of the procedure, as well as
how painful the last 3 min of the procedure were.
For example, take the following discomfort ratings
2-5-8-4, which indicate sequential periods in
minutes of aversive experiences ending with a
discomfort rating of four. Patients judged their
exposure to this series of discomfort as much less
of an uncomfortable experience than was the case
with the series 2-5-8, although the duration of their
exposure to discomfort in the 2-5-8 series was
shorter overall. The significant fact here is that
subjects underestimated the duration of certain
experiences and the effect of a more aversive
ending in a scenario made it just seem worse. This
result indicates that the evaluation of an aversive
episode could be improved by adding a period of
diminishing discomfort to the end of the
experience; whereby more pain is perceived as less!
In fact, Redelmeier et al. (1996) proved that the
above influences still held when the subjects were
given a direct choice of more pain over less;
unknowingly preferring an actually more painful,
but differently choreographed treatment scenario
– thus, confirming human PPR biases.
3. J/DM-PPR in co-design, the hypotheses
The important research question for HCI and
usability as targeted here is:
Can human J/DM-PPR bias effects be successfully
exploited for more meaningful usability evaluations
in various task and cultural contexts in
organisations as well as in the attendant change
management life cycle?
To investigate such research questions initially a
variety of instruments and protocols were
considered. These involved paper-based surveys,
video recording of task series, tape-recorded
interviews as well as designing and integrating the
PopEval online usability evaluation tool. Such a
tool was to provide a facility for periodic or event-
triggered online usability self-reports/mining,
running concurrently with the interactive system
such as the Web site that was being subjected to
 Online point-of-click Web usability with PopEval
Atta Badii
usability evaluation. Following some initial
experiments with run-time interviews/walk-
throughs and paper-based surveys, it was
concluded that these traditional HCI approaches
to capturing usability data, did indeed introduce
more constraints and distortions, as they were
more disruptive, more costly and less easily
manageable and certainly in no way as socially
scalable as using an online evaluation tool. Under
the above survey type protocols, subjects
registered a higher annoyance factor as they felt
repeatedly forced into a major focus switch between
the screen and a piece of paper on the desk, or the
interviewer; they found this too disruptive. Thus,
to study the hypotheses as stated below, it was
decided to use PopEval as an unobtrusive online
evaluation tool to keep tracking events, and/or
elapsed time, in attempting a least disruptive
capture of the most usability-critical data in near
real-time when it was likely to be most reliably
remembered and articulated. This research aimed to
test the following hypotheses:
H0. This stated that J/DM-PPR bias effects as
proven in other domains are also applicable
to the usability evaluations of interactive
software and in particular, to dynamic
usability evaluation of Web Mediated
Systems.
H1. This states that the above dynamic usability
theory lends itself to a more readily
computable analysis of transitional usability,
which can exploit the J/DM-PPR saliency-
recency type effects being investigated in
this research. Thus, we postulate a dynamic
usability process model, which can be stated
as:
UeT n / ðMP* oRÞT ðn21Þ
h i
/ F ðPCFÞT 0 ; . . .; ðPCFÞT ðn21Þ
h i
F ðPCFÞT 0 ; . . .; ðPCFÞT ðn21Þ
h i
/ F ðSIÞT 0 ; . . .; ðSIÞT ðn21Þ
This states that usability as evaluated at any time
(Tn) i.e. UeTn is a function F of the mutual
man-machine patterns-of-relating, i.e. (MP*oR)
as established at the previous instant T(n2 1); this in
turn depends on the value of the user’s PCF as a
global (aggregated) evaluation over the interval
[T0, T(n2 1)] and that such PCF is itself a function
of the salient impression (SI) formed by the
user about the system over the same period
(i.e. from the beginning to the previous instant).
The non-monotonic function F as applied here to
both SI and PCF denotes the non-linear
relationship that modulates the J/DM-PPR
294
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
theoretic global evaluations of a user’s affective
experiences over the transaction interval
(i.e. inclusive of natural biases over the scenario of
user’s various exposures to the system; one or a
series of service scenarios). Thus, H1 states that if
H0 were to hold, then this would imply the validity
of the above dynamic usability process model.
H2. If H0 were to hold, and, a tool such as
PopEval could perform online usability
evaluations, this should provide more
reliable usability-critical data intelligence,
more cost-effectively and more scalably than
manual paper-based methods, without
causing significant user disruption and/or
annoyance. (i.e. without the process
suffering the consequences of J/DM-PPR
distortions and the problems of the
disruptive usability survey methods).
In this work, we fully expected our usability
evaluations to provide finer-grain usability data
that was remedially prescriptive and effectively
actionable; captured with human distortions/
nuisance minimised; allowing the exploitation of
natural J/DM-PPR theoretic biases in evolutionary
systems co-design. The potential savings in the
dynamic maintenance of higher usability online
services and churn management motivated the
deployment of PopEval. This is crucial particularly
in highly volatile markets such as mobile telecom
and online media services but also generally in
technology acceptance by organisations as well as
in marketing of services and their uptake across
diverse cultures.
4. Methodology
Two mutually supportive methodologies
constitute the paradigmatic basis to this research.
Here, we first outline the methodological elements
of the C-Assure user-centred co-design framework
that inspired our research into the design of online
evaluation tools and J/DM-PPR. Next, we describe
our methodology in investigating the usability of
PopEval and the applicability of JD/M-PPR biases
to usability evaluation, i.e. in testing the
hypotheses H0-H2.
The C-Assure framework adopts a multilayered,
component-based approach to IS design,
development, diffusion and change management
(Badii, 1999, 2000; Badii and Rolfe, 1996; Badii
et al., 2000). It aims to integrate a toolkit of
methodologies for minimising IS failure risks using
smart project planning (re-adaptive, contingent
methods). C-Assure specifies a number of
infrastructural psychosocio-technical layers as
essential components for an IS planning,
development and diffusion support environment
 Online point-of-click Web usability with PopEval
Atta Badii
(IPDSE). These layers are to help provide high
transparency of alternative design and
development choices and their rationale; making
choices readily revisitable by all stakeholders
(Badii, 1999; Hounat and Badii, 1996). This is to
provide a capability for both short- and long-pan
organizational memory recall and re-learning from
IS intervention choices and outcomes. Thus, such
an IPDSE platform requires layers for contingent
(re)planning, latencies/epochs modelling and
benign control to help achieve culturally
comfortable IS-mediated enterprise
transformations. For example, the IPDSE must
offer reliable semiotics analysis and process/
cultural-match audit tools to enable the
interpretation and appropriate response to any IS
change-push, a market-led “IS Persona”,
compatible with the explicit or implicit change-
pull from users towards their own future user-led
“Group Persona” or collective wish for process/
cultural change (Badii, 1999). This is predicated
on the expectation that IS development should
promote user ownership, actability,
empowerment, mutually beneficial balances of
power and co-operative patterns-of-relating within
the system (man, machine, process) as a whole
(Badii, 1998; Badii and Rolfe, 1996).
In this way C-Assure supports the
organizational strivings for such ideals as
emancipation from unproductive processes and
patterns of power towards constructive and
profitable P*oR. However, C-Assure extends this
support also to man-machine or user-system
relationship management (Badii and Rolfe, 1996).
Accordingly within this framework, the mutuality
assumption is that both types of transactors, the
IS, and the user, should offer, and expect to
receive, co-operative patterns-of-relating, and
discourse modalities to support constructive
dialogue in user-centred co-design (Badii, 1999).
Each transactor (i.e. user or system) should
attempt to offer some of the preferred comfort
modalities of the other transactor
(i.e. accommodations), whilst expecting some
reciprocal effort by the other transactor to reach
out to cope with such comfort modalities which, in
this way, emerge as realisable by it within system/
user’s resource/cognitive constraints i.e. within
their reachabilities (Badii, 1999, 2000).
The distinctive mindset here is the insistence on
man-machine mutuality of intelligibility and
scanability (accessibility, transparency) to allow
meaningful dynamic assessment of usability,
strategic-fit, process-fit, power balance, privacy/
identity/self-disclosure rights, and benefits, for all
stakeholder relationships within the entire
organisational setting – viewed as a meta-system
295
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
promoting user-centred co-design of systems
(Badii, 2000; Badii and Rolfe, 1996).
C-Assure views the psychosocio-technical
acceptance and thus, sustainable adoption of such
a meta-system as depending critically on two
properties:
(1) the level of sustainable system
accommodation of the target user’s important
and deeply valued needs afforded by the new
IS through its design features.
(2) the expected level of the adaptation potential
of a target user group – reachable without
them suffering perceived intolerable cultural
stress/distress (user’s reachabilities).
Thus, the ideal IS would aim to reduce the level of
man-machine P*oR imbalance or cultural/process
mismatch by attempting to strike realisable and
affordable co-design trade-offs for the whole
situated meta-system. Accordingly, the IPDSE
should provide cost-effective access to meta-
models, tools, repositories, ontologies and
business/process logics to support mass
re-negotiation of stakeholder needs; not confining
user involvement to “focus groups” only. This
will significantly aid a sense of ownership born of
co-operative sense making and widely shared
understandings in co-design, and thus, more likely,
a higher PCF and cross-cultural interoperability of
the resulting system.
Naturally, C-Assure relies on an ontological
framework to serve contingent co-design method
deployment in dynamic situated contexts. This
requires a pattern language representation of
suitable system development methods and their
sub-methods, triggers and effects; referred to as a
Co-Design methodologies framework (Badii,
2000). C-Assure demands that the “meta-system”
undertakes only smart projects capable of exploiting
the above layers whilst executing sets of
conditional co-design plan segments supported by
a framework for just-in-time, least-committed and
anticipatory usability knowledge exchange. This
would allow functions such as usability mining,
impact/risks modelling, serendipity management,
degradation tracking and anticipation. IS
interventions, managed as smart projects, need to
foresee and manage the effects, side-effects and affects
that follow each design choice that they make
throughout their life cycle; from inception through
post-deployment maintenance. Such smart
projects need these capabilities for failure
pre-emption/recovery and time-critical risks/
resources balancing.
Clearly the required knowledge basis for a
C-Assure smart project framework would include
design patterns and their contingent selection rules
(e.g. situated contraindications, confluences,
triggers and consequences, etc). Another
 Online point-of-click Web usability with PopEval
Atta Badii
infrastructural layer would perform a situated
semiotics appraisal and the modelling of
comfortable cultural transformation trajectories
for workgroups. The support environment also
includes a model of requirement elicitation tools
and their applicability criteria including a library of
elicitation scripts to support the dynamic enquiry
process. Negotiation of users’ current needs is thus
facilitated by the elicitation of a consensus
specification of collective and personal
compromises about system features. The
justification roots for the above compromise sets,
version (sub)spaces, and the routes whereby such
versions were arrived at can be made transparent
and readily re-examinable by using a framework
such as the multimedia requirements engineering
assistant, MMREA (Hounat and Badii, 1996).
Accordingly, C-Assure is deployed to achieve the
best current trade-off between realisable system
accommodations of users’ chief needs, and user-led
relaxations of selected constraints through
compromises on secondary user benefits (semantic-
cooperative negotiations and discoveries). In this way
systems rethinking in C-Assure is fundamentally
based on providing the infrastructural layers to
support an ongoing psychosocio-technical multi-
logue and maximising the transparency of the
information relevant to user-centred co-design
(the roots and routes to current specification); as in
boundary sensitised information relationship
management for mutual benefit, i.e. BSIRM_MB
(Badii and Rolfe, 1996).
This approach is thus, strongly wedded to
maintaining socio-technically inclusivist and
dynamic information accessibility patterns within
and across sub-cultural groups. Full transparency
of the justification hierarchies for IS co-design
trade-offs and intervention choices is a prerequisite
for this. The transparent balancing of affordable
system accommodations with expected user’s
reachabilities helps C-Assure achieve and sustain
usability and cultural/process matchmaking. Over
time this allows the organisation to update the
repositories in the Co-Design methodologies
framework, to allow learning from experience i.e. to
know what type of IS-mediated change would prove
socially/culturally scalable and comfortable for a
particular sector or sub-group. In this way, user-led
cultural/process migration wishes can first be made
explicit, using the online negotiation tool, and then
such wishes can be aided by consensual IS
interventions that are in tune with the user
environment and respect the limits of users’ cultural
elasticity and their goals. Implicitly the C-Assure
research focus is to support dynamic user-centred
co-design through (nested) spiral models of system
development led by dynamic (online) assessment of
fit. This is best achievable through the routinisation
296
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
of an interpretivist-iterative basis for mass-
consultation, exploration and analysis of competing
IS design routes and their situated impact. This will
be greatly aided by the integration of the IPDSE
with a business process re-engineering simulation
platform. Such an integrated system will aid
efficient and dynamic projection of IS development
paths using a managed mix of development models
and methods. Such a framework for visualisation
and renegotiation can be used to aid cost-effective
matchmaking and failure avoidance/recovery in IS-
mediated change management. This approach
would be particularly recommended in first-mover
innovation management, as in IS-mediated
enterprise transformations for efficient integrated
electronic and mobile commerce (e/m-commerce).
5. Experimental procedure
To investigate the hypotheses (H0-H2), a
methodology of simulation and testing was
conducted through an empirically grounded study
involving multi-modal and multimedia usability
data capture and triangulation of results. This used
PopEval (Badii, 2000) in the context of both Web
and non-Web application domains in parallel
usability experiments as follows: the Web site
applications comprised 15 major online interactive
shopping sites and the non-Web application was a
typical interactive desktop publishing tool
(MS PowerPoint). The two tasks, desktop
publishing and online shopping were chosen
because they constituted a reasonable and
manageable set of challenges familiar to most users
irrespective of their level of everyday exposure to
computing.
A stratified random sample of 127 participants,
including both computer/Internet savvies, and,
naive users of both genders covering the age
groups 20-70, were recruited. All 127 users filled
two separate questionnaires each. One set of pre-
experiment questionnaires allowed us to plan the
conduct of the experiment and the balancing of the
mix of users for each experiment through prior
knowledge of each user’s background and their
level of experience with computing in general and
with online Internet shopping in particular.
Accordingly, we ensured that a wide variety of
goods, for example, music, books, clothing, and
food, were available from the Web sites chosen for
the experiment; to include the type of items each
user would be most interested in buying online.
The second set of questionnaires, supplemented
by a video-recorded interview, post-experiment,
were to provide an additional instrument for
discovering users’ experiences during their
respective sessions doing online shopping or using
the desktop publishing software. This was done so
 Online point-of-click Web usability with PopEval
Atta Badii
as to provide triangulation with the results from
the automated online usability evaluations carried
out by PopEval. Of the 127 subjects, 62 took part
in usability evaluation experiments using the
Web sites and 65 participated in usability
experiments for MS PowerPoint. The 62 users
participating in the online shopping experiment
had a wide choice of Web sites, including their
preferred retailer sites, all preset in the browser as
favourites. This allowed the users to be just a click
away from leaving or resuming their interaction
with any online shopping Web site whilst
attempting to spend their nominal budget. Of this
group 22 were video recorded using various online
shopping sites with a time limit of 30 min each and
another limit of £100 nominal budget to spend.
Whichever limit to be reached first terminated
their online shopping after which they filled their
post-experiment questionnaire followed by a
video-recorded interview. At the start of all video
recording sessions, the users had been encouraged,
but not forced, to think aloud about, i.e. verbalise,
their intentions, actions and experiences, pleasant
or otherwise; all the video-taping sessions were
separately audio-taped for transcription purposes.
As an experiment, the online shopping sessions
did not involve actual payment procedures as once
a user had completed the task of selecting and
inserting an item into their shopping basket and
the item was deemed to have been purchased
and its price was deducted from the user’s budget.
An indication of how much money and shopping
time was left to be spent by each user was made
available on a separate monitor, located close by,
to enable each user to check their own progress at a
glance whenever they wished. These 22 online
shopping sessions were video recorded in dual
channels, using two cameras for each session;
thereby recording both user’s face, down to their
shoulders, as well as their monitor screen
simultaneously throughout each session.
Subsequent video processing yielded a
synchronised split screen video recording showing,
on one screen, both user’s facial expression and
their utterances, on left half of the screen, with the
current Web page and the user’s mouse
movements over it appearing on the right half of
the screen – from the first click to the last.
After each user’s online shopping session,
following a simple distractor task, the users were
invited to take part in what we termed NVUs:
i.e. “nested video usability data capture sessions”.
These entailed each user taking part in a video-
recorded interview whilst watching the above split
screen video recording of their own earlier 30 min
session, thus re-living their experiences by talking
through their feelings about system usability whilst
being able to point at each click and each screen
297
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
element or pause each frame at any stage along the
session as their 30 min split screen video played
out. Thus, throughout this nested-video interview
the earlier video recording of the sessions was
being used as a reminder in seeking to elicit specific
situated evaluations given user’s affects/salient
impressions arising from user’s latest experience
with the Web sites services. The primary video
recording round resulted after 22 h of video
recordings; reduced to 11 h after split screen
compilation. The subsequent 22 nested video
recording sessions generated a further 19 h of
secondary video recordings. Thus, the resulting
total of 41 h of video recorded data together with
its audio track of user verbalisations constituted a
very rich data source lending itself, as it did, to
situated interpretations by means of conversation
analysis, gesture analysis, transaction analysis, and
attitude analysis. Additionally the transcriptions of
the audiotapes were subjected to extensive
discourse analysis. Another 40 users carried out a
similar online shopping exercise under the same
protocols of 30 min time limit and £100 nominal
budget limit, except that here instead of the video
recording, PopEval was being used to record data
from each user regarding their intentions, actions
and experiences.
The remaining 65 users completed a desktop
publishing task using MS PowerPoint with
PopEval running in the background so as to
provide automated usability elicitation and
evaluation. These users had experience of using
various types of software on the same PC platform
as used in the experiment (Windows 98/NT
systems). In this way the users would have the
necessary basic experience of the computing
environment and sufficient familiarity with the
operating system. The only other requirement was
that those taking part in this phase of the
experiment had not used earlier the software
package, MS PowerPoint, which was to be
evaluated, i.e. they were novices as far as MS
PowerPoint was concerned.
Each user’s workstation was located in an
informal setting so that they would be as relaxed as
possible. Again to ensure that there was an
appropriate context, as a “productivity baseline”
for the exercise, to help relate user’s perception of
personal progress and goal-driven focus, they were
all supplied with an instruction sheet explaining
what actions they were expected to attempt to
perform using the MS PowerPoint during the
30 min session; together with a guide sheet on the
use of MS PowerPoint. The PopEval tool then
asked the subjects to enter their pre-allocated
userID, gender and age. They were then told to
launch MS PowerPoint and to attempt the tasks
outlined on the instruction sheet requiring them to
 Online point-of-click Web usability with PopEval
Atta Badii
perform a set of typical desktop publishing actions
with increasing degrees of challenge; using the MS
PowerPoint Package to the full. Prior to the session
the PopEval experiment editor had been used to
plan, edit and set-up a questionnaire file for the
PopEval to run during the experiment. The users
had been informed that they would be invited by
PopEval to make their feelings of (dis)comfort with
the experiment known by selecting a value on a
scale, in the PopEval window, at certain points
during the exercise, if they wished. For this
experiment at a set interval of 6 min the subjects
were asked for their optional evaluations using the
following request: “Please select a value on the
scale below to indicate your level of comfort/
satisfaction and type a word or a phrase to give any
reasons”. The Likert scale that appeared below
this request, in the PopEval window, displayed the
following anchor labels: left anchor label:
“Content”, right anchor label: “Unhappy” and it
was thus possible for users to indicate their level of
(dis)satisfaction along the scales. User’s selection
of a position along this interval scale was also
supported by an animation of increasingly happier
or unhappier faces and colour changes in
sympathy with user’s movement of the cursor
along this “thermometer scale” up or down from
one extreme to another (using direct sensing/
manipulation modes). The users could thus easily
indicate how they felt about any given usability
aspect being probed in the respective questions.
Immediately after the attempt of last set tasks, the
users were asked to close down MS PowerPoint
and leave the computer system. A distracter task
was then used to take the users’ minds off the
experiment. Thereafter, all the users were
provided with two requests for retrospective
evaluations to be made by them with reference to
appropriate interval scales as follows.
(1) “Please give your overall (global) evaluation of
the time you have spent as a participant in this
experiment, indicate the level along the scale
as appropriate, the leftmost rating counts as
pleasant and the rightmost as unpleasant”.
(2) “Please give your overall opinion of the
PopEval tool where 0 stands as not annoying
and 10 as very annoying”.
This last scale was used to obtain users’ assessment
of the annoyance ratings for PopEval.
6. Results and conclusions
The results of this research came in three forms.
The first type of results related to H0 and H1,
i.e. the J/DM-PPR biasing effects and thus, the
validity of the new dynamic usability definition and
298
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
its process model as postulated in this work. The
second set of results concerned the validity of H2
i.e. the performance of the PopEval prototype,
which was judged satisfactory as will be described
later. Specifically, the results from the tests in this
study were found to support the findings from
recent cognitive psychology research on J/DM-
PPR theoretic biasing effects including “duration
neglect” bias and the “peak-and-end evaluation”
bias. Here too, our results showed that the global
evaluation was not linearly dependent on (e.g. was
not a statistical mean or a median of) the real-time
evaluations as might have been suspected. Some
kind of non-monotonic “judgement process” was
having an effect on the global (aggregated)
evaluations given by our users, just as had been
predicted in the studies by J/DM-PPR researchers,
as detailed earlier. The results also showed only a
marginal difference between the last real-time
evaluation and the global evaluation, again
confirming the expected influence of this
particular recency or “end-effect” type of human
recall bias. Thus, our results confirmed the validity
of J/DM-PPR bias theories in user’s perception of
system usability; i.e. both dynamic theory
hypotheses H0 and H1 were proven.
As for H2, the results of the annoyance ratings
for PopEval amongst this mixed user group of 127
were very similar ranging from two to three within
a 0-10 interval scale. This indicated that our
PopEval evaluation tool itself did not cause a
significant distortion effect, unlike paper-based
and interview surveys that were found to do so.
Thus, this supported the H2 hypothesis.
Regarding MS PowerPoint (the software being
evaluated) the worst ratings were all given in the
first 6 min of the experiment. This could be taken
to suggest that for the users in this study MS
PowerPoint tended to appear relatively difficult
initially, but once they had got past the first few
steps they found it relatively less problematic to
explore and use in order to perform their tasks.
Alternatively it could be said that all users tend to
find the steep start-off period of (re) familiarising
themselves with an interactive software system, say
the first 10 min, a relatively more challenging
period.
The PopEval experiments produced a wealth of
other results regarding purchase-specific triggers
in the Web space, as did the NVU protocol.
A notable example, however, of how the online
shopping experiments supported the hypotheses
was the frequent observation that once a shopper
had happily located a desirable item then they
tended to be relatively less affected by the minor
difficulties in locating the shopping basket and
completing the purchase. For example, finding a
much desired item such as the latest sportswear
 Online point-of-click Web usability with PopEval
Atta Badii
affected a user’s “salient impression” so positively
that even a subsequently arduous search to
eventually locate a “not-so-well-designed
shopping basket” was tolerated well, provided that
they were able to effect “the purchase” in the end
i.e. drop the item in the basket. The search to
locate the shopping basket each time became less
uncomfortable than the intervening difficulties in
finding a desired item; so was relatively forgotten/
forgiven particularly as it was superseded by a
satisfactory closure, i.e. finding the desired item
and completing its purchase by dropping it in the
shopping basket. The results of this study offer
immediately available insights applicable to
usability research, and indeed to other domains as
well such as business process re-engineering,
eCRM/churn and change management.
The importance of the above findings for
software systems, and in particular, for interactive
Web site development, is that if the interactive
features which tax the user’s patience and cognitive
capability are not severely restraining and the task
closures, particularly, are “reasonably smooth”,
then users will be left with a better impression of
the usability of the system than would be the case
otherwise. As for the optimisation of user-centred
Co-design, the results enable significant insights to
be gained about how to assess where and what
design trade-offs most matter to users (i.e. spot the
dynamically more memorable system affordances
over the service scenario). They also provide
insights about the circumstances whereby users
would be happier to forego certain system
capabilities (i.e. spot the dynamically less
memorable system affordances over the service
scenario) provided certain other more deeply
valued or well-placed system capabilities can be
available.
In short dynamic usability evaluation can be
exploited to satisfy users more cost-effectively by
identifying those interactive features that matter
most within evolving sector-specific service
scenarios and those features about which users can
become more forgiving thus making them
relatively less usability-critical.
The most valuable insight is however that the
issues that are the key determinants of user
(dis)satisfaction, and thus, productivity, are indeed
dynamic in nature. That is, in the highly dynamic
open-world of ubiquitous accessibility such as
www, usability evaluation based on traditional
static HCI guidelines can be powerfully supported
with dynamic usability models to allow a more
meaningful evaluation of user experiences and the
current status of the evolving stakeholder usability
relationships. In this way dynamic usability theory
will help integrate results from the established
psychosocio-technical models to allow inferences
299
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
about the more specific causes of user
(dis)satisfaction in the evolving user-system
relationships. This can cost-effectively inform the
collaborative usability choreography (co-design) of
affordable service features for the highest possible
perceived usability and process-fit for given user
groups – to the mutual satisfaction and benefit of
all stakeholders.
The “peak-and-end evaluation” and “duration
neglect” are the two most significant insights to
have immediate exploitation potential in software
engineering and in particular, in the design of Web
Systems and Services such as Virtual Shopping
Malls, etc.
Online self-report and evaluation tools such as
PopEval as a component of a user-centred
Co-design methodology such as C-Assure, can be
run from an evaluation server for asynchronous,
online mass-consultation supporting the design of
dynamically personalised user-intimate systems.
This facilitates iterative renegotiation of all
stakeholder needs whilst minimising nuisance and
bias. It will ensure efficient and effective
attainment of high usability, user-intimate systems
which remain aware of dynamically usability-
critical features to be accommodated over given
(sectorial) service scenario for given classes of
users or individuals. This is to serve the new era of
ubiquitous user-intimate intelligent media systems
and services that is now looming on the horizon.
References
Ahlawat (1999), “Order effects and memory for evidence in
individual versus group decision making in auditing”,
Journal of Behaviour and Decision Making.
Badii (1998), “Information relationship symmetry management
for success in outsourcing”, Proceedings of Americas
Conference on Information Systems Annual Conference,
AMCIS98, August 1998, Baltimore.
Badii (1999), “A framework for cultural accommodation with
sensitised systems for usability relationships evaluation
(C-Assure)”, UKAIS Midlands Research Seminar Series,
De Montfort University, October 1999; abstract in
Summary Proceedings of UKAIS (Midlands), UKAIS
Newsletter, June 2000.
Badii (2000), “Dynamic Usability Mining with PopEval and the
C-Assure Methodology”, EnCKompass Internal Report
2000-MM2, EETDT, December 2000.
Badii and Rolfe (1996), “Boundary sensitised information
relationship management”, Proceedings of the 1st United
Kingdom Academy of Information Systems Annual
Conference, UKAIS’96, April 1996, Cranfield University,
Cranfield.
Badii et al. (2000), “Online usability evaluation and content
management for mass-personalisation in e-everything”,
Proceedings of the 5th United Kingdom Academy of
Information Systems Annual Conference, UKAIS 2000,
April 2000, Cardiff University, Cardiff.
Dix et al. (1993), Human-Computer Interaction, Prentice-Hall,
Englewood Cliffs, NJ.
 Online point-of-click Web usability with PopEval
Atta Badii
Draper et al. (1996), “Integrative evaluation: an emerging role
for classroom studies of CAL”, Journal of Computers and
Education, Vol. 26, pp. 1-3.
Hounat and Badii (1996), “Multimedia requirement elicitation
assistant (MMREA)”, Proceedings of the 1st International
Conference on Knowledge Transfer, KT’96, July 1996,
London University, London.
Kahneman et al. (1982), Judgement Under Uncertainty:
Heuristics and Biases, CUP, USA.
Kahneman et al. (1993a), “Duration neglect in retrospective
evaluations of affective episodes”, J. Pers. and Soc. Psych.,
Vol. 65.
Kahneman et al. (1993b), “When more pain is preferred to less:
adding a better end”, J. of Psyc. Sci., Vol. 4, p. 6.
Lange-Kuettner and Martin (1999), “Reversed effects of
familiarity and novelty in visual and auditory working
memory for words”, Journal of Brain and Cognition,
June 1999.
300
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 291–300
Mahrer and Miles (1999), “Memorial and strategic determinants
of tactile recency”, J. of Exp. Psych., May 1999.
Redelmeier et al. (1996), “Patients’ memories of painful medical
treatments: real-time and retrospective evaluations of two
minimally invasive procedures”, Journal of Pain, Vol. 1
No. 66, pp. 3-8.
Shneiderman (1982), “The future of interactive systems and
emergence of direct manipulation”, Journal of Behaviour
and IT, Vol. 1, p. 3.
Further reading
Knoedler et al. (1999), “The shift from recency to primacy with
increasing delay”, J. of Exp. Psych, March.

Other papers
Benefits and barriers of
electronic marketplace
participation: an SME
perspective
Rosemary Stockdale and
Craig Standing
The authors
Rosemary Stockdale and Craig Standing are based at the
School of Management Information Systems, Edith Cowan
University, Perth, Australia.
Keywords
Small to medium-sized enterprises, Electronic commerce,
Trade barriers
Abstract
There are concerns that despite government initiatives to
promote adoption of electronic commerce, SMEs still fail to
realise e-commerce related benefits. It may therefore, seem
premature to discuss electronic marketplaces in the context
of SMEs. However, if SMEs ignore e-marketplaces a number of
problems can result. E-marketplaces present a significant threat
to SMEs since they increase competition and leave
non-participants vulnerable to more e-enabled firms. This paper
examines the barriers and benefits of e-marketplace
participation by SMEs. The nature of e-marketplaces is addressed
and the benefits of participation are examined. Drawing on the
literature, the barriers facing smaller firms in this environment
are discussed. Identification of these barriers, such as lack of
standards, supply chain integration and global trading, enables
a greater understanding of how SMEs can plan effective
strategies to gain from e-marketplace participation.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 301–311
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548715
301
Introduction
Small to medium-sized firms (SMEs) make
substantial contributions to national economies
(Poon and Swatman, 1999) and are estimated to
account for 80 per cent of global economic growth
( Jutla et al., 2002). However, despite a wealth of
initiatives from governments, significant concerns
remain as to how smaller businesses can benefit
from the electronic environment (van Akkeren and
Cavaye, 1999; Korchak and Rodman, 2001; Lewis
and Cockrill, 2002).
Although the availability and falling costs of
personal computers have had a major effect on the
ability of SMEs to compete in electronic
commerce (Cragg and King, 1993; Poon and
Swatman, 1999), the impact of this has been
a mixed blessing for many firms. The breadth and
speed of the changes brought by the Internet
have radically altered the business landscape.
Firms need to plan effective strategies to realise
benefits from the dynamic and information rich
environment (Downes and Mui, 1998).
The electronic environment can be intimidating
to many smaller firms and the development of
information systems such as electronic
marketplaces remain a mystery to many SMEs.
It may seem premature to discuss e-marketplace
adoption by SMEs since many SMEs are not yet
secure with other, perhaps simpler, applications of
e-commerce. However, there are problems
associated with a decision to ignore e-marketplace
opportunities. Larger organisations, with their
more extensive resources, are looking to
e-marketplaces to take advantage of the significant
trading benefits offered and anticipate that
suppliers will be available through such
marketplaces. The ability of such marketplaces
to facilitate trading over regional and geographic
boundaries at low cost and without regard to the
size of the firm opens up all markets to broader
competition. While this supports firms in seeking
new markets, it also leaves traditional markets
open to outside competition. In short, the problem
for SMEs is not just one of lost opportunities but
encroachment on their markets in a fiercer,
competitive environment.
The contribution of this paper is to define the
potential benefits that can be obtained by SMEs
that participate in e-marketplaces and to examine
the barriers that smaller firms need to overcome to
benefit from participation. SMEs in the context
of this paper are defined according to the
Australian Bureau of Statistics as firms employing
less than 200 full time equivalent workers and
are not subsidiaries, public companies, or
incorporated bodies (ABS, 2003). SMEs need
to understand e-marketplace opportunities to
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
develop informed strategies (Brunn et al., 2002).
This will enable decisions regarding participation
to be taken more effectively by smaller firms
seeking to extend their markets, to retain contracts
with larger organisations or to deal with
competition from non-traditional competitors.
Electronic marketplaces
The existence of marketplaces in human society
has a long history from before the Agora of
Ancient Greece to the online trading places of the
21st century. Trading of goods and services for
other goods or for money is central to the concept
of human socialisation (McMillan, 2002).
The advent of the electronic environment has not
changed the principles of markets and marketplace
trading, merely the way society goes about trading.
In essence, the technology facilitates the business
of the market, but it is not the reason for the
market to exist. Online markets must offer an
advantage over traditional markets if they are
to succeed and encourage firms to overcome any
difficulties arising from using the technology.
Consequently, an online market must be as rich,
complex and complete as a traditional market and
must create extra value for its users (Kambil and
van Heck, 2002).
The development of electronic marketplaces
followed swiftly on the use of the Internet for
business purposes. The initial proliferation of
e-marketplaces proved to be unsustainable and
a forecasted period of consolidation is now
underway (Forrester Research, 2000).
The number of marketplaces in any one industry
sector has been considerably reduced and the
methods of transacting business and generating
revenue have matured. In addition, the scope of
the value added facilities has increased and
become more targeted to the market. In this
evolving environment there is some confusion as to
what constitutes an electronic marketplace. Bakos’
(1991) early definition of “an interorganisational
information system that allows the participating
buyers and sellers to exchange information about
prices and product offerings” has been widely
accepted (Choudhury et al., 1998; Clemons et al.,
1993; Forrester Research, 2000). This definition
has been refined and updated to take account of
the changes in the e-marketplace environment,
and the terminology, that have arisen from
developments of the World Wide Web. Terms such
as e-hubs, portal, exchange and auction are used in
different contexts with contradictory meanings
assigned to them. To avoid confusion, but with due
acknowledgement of the complexity of the
environment, the paper incorporates key elements
302
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
of several definitions (Federal Trade Commission,
2000; Grieger, 2003; Raisch, 2001; Sculley and
Woods, 2001; Weill and Vitale, 2001). Based on
this an e-marketplace is defined as:
An interorganisational information system that
allows multiple buyers and sellers, and other
stakeholders, to communicate and transact
through a dynamic central market space, supported
by additional services.
The structure of electronic marketplaces
Despite the differences and wide variety in market
makers’ business models three main elements in
the structure of an e-marketplace are identified
by the literature:
(1) the origins of the marketplace, who owns and
operates it;
(2) the transaction mechanisms that are offered
by the marketplace; and
(3) the additional facilities that a marketplace
offers to its participants.
Ownership models
Ownership models have become more diverse as
the number of marketplaces have increased and
market makers have reviewed and refined their
business models. Four identifiable marketplace
structures have been recognised.
Intermediaries
Intermediary marketplaces, one of the earlier
models of ownership, operate services across
industry sectors concentrating on delivering
generic services such as auction facilities or
value add services (Skjott-Larsen et al., 2003).
They bring together buyers and sellers to allow
trading to take place through a variety of
mechanisms (Figure 1).
Consortia
The intermediary marketplaces were soon
followed by large multinational organisations
investing in partnerships to form unprecedented
collaborations with competitors to launch
industry specific marketplaces (Raisch, 2001).
The structure of the consortium style marketplace
gives advantages to the owners by providing a focal
trading point to attract suppliers to a specific
industry. Suppliers’ advantage lies in access to
supply chains of large organisations (Figure 2).
Hierarchies
A more recent development has been the increase
in the number of private marketplaces or
hierarchies. Large organisations invest in and host
their own marketplaces thereby retaining control
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Figure 1 Intermediary style structure for an e-marketplace
Figure 2 Consortia style structure of an e-marketplace
of the facilities they develop and offer to
suppliers (Bar, 2002). Such marketplaces require a
large investment of time, money and technical
expertise and are beyond the scope of many
organisations. However, the development of
off-the-shelf software may enable smaller
companies to launch less complex versions
(Figure 3).
Included in the hierarchical model is the
increasing number of government e-marketplaces.
These fall into two categories: e-procurement hubs
for government and government hosted sites to
support and encourage e-commerce.
Cooperatives
The cooperative or large group ownership
structure of e-marketplace is anticipated to arise in
the near future. This model is based on a group of
stakeholders cooperating as market makers for
common interest. Common interest may lie within
the type of industry, a geographic area, or a specific
goal (Standing et al., 2003) (Figure 4).
303
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
Transaction mechanisms
There are several different transaction
mechanisms evident in electronic marketplaces,
the most common being online catalogues,
auctions, negotiation facilities and exchange.
There is a great variety within each of these
mechanisms. For example, over 30 types of online
auction have been identified (Davis, 2001) and the
types and features of catalogues are numerous
(Stanoevska-Slabeva and Schmid, 2000).
Currently, there is little empirical evidence of the
optimum type of mechanism for different goods
and services, although categorisations of
e-marketplace types have been made (Kaplan and
Sawhney, 2000; Sculley and Woods, 2001).
Some market makers specialise in one
mechanism, for example, FreeMarkets
(www.FreeMarkets.com) specialises in auctions
while others offer a range of mechanisms.
For example, Quadrem (www.Quadrem.com)
offers negotiation, catalogues and auction.
The exchange mechanism tends to be more
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Figure 3 Hierarchical style structure of an e-marketplace
Figure 4 Cooperative ownership style structure of an e-marketplace
prevalent where established exchanges have
established a Web presence, such as the London
Metals Exchange (www.lme.co.uk).
Additional facilities
Market makers use value-add facilities to enhance
the attractiveness of their Web sites in pursuit of the
competitive advantage necessary to survive in
the developing environment of e-marketplaces
(Bakos, 1991). The facilities on offer range from
information services, such as a listing of industry
events, research papers, tutorials and news, to
transaction orientated facilities. These include
insurance, online payment, escrow, completion of
304
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
customs paperwork, data warehousing and
tracking of deliveries. E-marketplaces can
contribute to the enhancement of trust by offering
verifications services and screening of potential
trading partners (Choudhury et al., 1998). Some
market makers adopt the community site model,
well described by Hagel and Armstrong (1997),
offering a complete range of facilities relevant to a
specific industry or regional area. Market makers
who have a good understanding of their target
participants develop market models to enhance the
value proposition for marketplace members.
Value-add facilities play an important role in
creating such models.
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Income models
An additional factor that may influence the
selection of an e-marketplace by any prospective
participant is the method of income generation.
There are several methods of income generation
including transaction fees, licence fees, advertising
and the sale of marketing data. The most common
method is the transaction related fee where buyers
or suppliers pay a percentage of the transaction
value to the market maker (Kambil and van Heck,
2002). Licence fees are less attractive to smaller
firms who are reluctant to commit to an untested
trading platform. These two methods of income
generation are seen to deter buyers and sellers from
entering marketplaces and this is influencing
market makers toward generating income from
service fees on value-add facilities where
participants are seen to be more willing to pay
(Federal Trade Commission, 2000; Kambil and
van Heck, 2002).
The benefits and barriers in
E-marketplaces
SMEs seeking to develop a strategy to support
participation in electronic marketplaces require an
understanding of both benefits to be gained and
barriers to be overcome. This section of the paper
examines the literature to identify the benefits and
barriers.
Benefits of E-marketplace participation
The advantages of e-commerce participation for
SMEs relate to their ability to keep pace with
a changing business landscape. Brought about by
information technology (IT), these changes
include facilitated access to global markets,
changed production methods and costs, enhanced
communication, reduced transaction costs and
stimulated competition (Sculley and Woods, 2001;
Timmers, 1999; Tumolo, 2001). The size of the
firms enable SMEs to be more adaptable and
responsive to changing conditions than large
organisations (Walczuch et al., 2000) and to
benefit from the speed and flexibility that the
electronic environment offers.
The new environment promises much to SMEs
from e-commerce, but adoption levels remain low
(Levy and Powell, 2003; Mehrtens et al., 2001;
Poon, 2000). This is partially due to lack of
understanding of the benefits SMEs can achieve
(Goode, 2002) and of unrealistic expectations of
benefits and the difficulties of evaluating them
(Poon, 2000). In contrast, the literature on
electronic marketplaces does not differentiate
between size of firm, but rather takes a “one size
fits all” approach to benefits (Fariselli et al., 1999).
305
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
Benefits that can be of relevance to SMEs are
identified, and are given in Table I.
Overall, the range of benefits that can be
achieved from participation in e-marketplaces is
extensive, although not all will apply to every
company. SMEs can realise benefits but it may
take time for them to be recognised within the
company. A steep learning curve will precede any
benefit gain for many companies and it should
not be expected that this can be achieved in
the short-term. A longer, slower approach may be
a more reliable way to achieve sustainable
advantages from e-marketplace participation.
Nevertheless, recognition of the advantages to be
gained from e-marketplaces is desirable from an
early point of developing e-commerce to enable
business strategies for selection of suitable
electronic marketplaces to be put in place.
Barriers to E-marketplace participation
Many SMEs are not achieving even minimal levels
of e-commerce adoption raising concerns as to why
adoption programmes, many of them government
led, are not more successful ( Jutla et al., 2002;
Korchak and Rodman, 2001; Van Beveren and
Thomson, 2002). Major barriers to increasing
adoption remain: lack of resources and knowledge
(Cragg and King, 1993; Mehrtens et al., 2001), the
skill levels of business operators (Darch and Lucas,
2002; Duan et al., 2002), lack of trust in the IT
industry (van Akkeren and Cavaye, 1999; Bode and
Burn, 2002), and the lack of e-commerce readiness
in some industry sectors (Lewis and Cockrill,
2002). A further barrier is the lack of recognition
of the potential to improve business appropriate
to the effort and costs of adoption and lack of
understanding of the realisable benefits
(Goode, 2002; Poon, 2000).
Those SMEs that have overcome these barriers
and started along the road to online business often
remain reluctant to move into the electronic
marketplace environment. The evidence for this is
being increasingly reported in the business press
(Erbschloe, 1999; Howarth, 2002), but remains
largely anecdotal. However, research into the
effects of global markets and supply chains is
beginning to examine more aspects of SMEs’
involvement in e-marketplaces (Fariselli et al.,
1999; Gulledge, 2002; Stockdale and Standing,
2003a).
The barriers to e-marketplace participation
sometimes reflect the more generalised barriers of
e-commerce adoption, but are discussed here
specifically in relation to the marketplace.
Lack of support from market makers
Market makers often aim their marketing at
the larger corporations and do not perceive
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Table I Benefits to be gained from SMEs participation in e-marketplaces
Advantage
Access to a wider range of markets
Greater potential for partnerships
Flexibility in administration and
communication
Convenience (24/7 accessibility)
Information
Improved customer services
Updating of information
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
Benefits Source
For suppliers there is the potential to Brunn et al. (2002), Essig and Arnold
broaden the company’s target market (2001), Fariselli et al. (1999), Senn (2000)
globally by seeking out marketplaces and Tumolo (2001)
with a global reach
For buyers there is potential to widen the
supplier base to find lower prices or new
product line
Electronic communication enhances the Hurwitz (2000) and Tumolo (2001)
ability to maintain geographically distant
relationships through e-mail and
multimedia programs, thereby widening
support for the supply/seller base
The use of the electronic environment Brunn et al. (2002), Hermanek et al.
enhances the flexibility and accuracy of (2001) and Hurwitz (2000)
administration procedures and facilitates
communication within a company and
across partnerships
Convenience in interaction with partners. Deeter-Schmelz et al. (2001), Hurwitz
For example, time zones are less (2000) and Lin and Hsieh (2000)
problematic when communicating
electronically and customers can submit
orders in their own time.
(It should be noted that a number of
issues such as timing for global auctions
require some concessions to normal
working hours)
An advantage of many e-marketplaces is Bakos (1998), Brunn et al. (2002), Burton
the accumulation of information into one and Mooney (1998), Essig and Arnold
site. It is in the interest of both market (2001), Lin and Hsieh (2000) and Weill
maker and participants that all parties are and Vitale (2001)
well informed, although a level of trust in
the marketplace must be established to
maintain confidence in the sources of
information. In addition, information
exchange is enhanced through the
offering of multimedia applications for
marketing, tendering, and design
purposes. Designs and plans can be
presented via site for tendering purposes
using software drawing packages. Some
sites offer Web services to develop
marketing for their participants
The ability to tailor customer services to Bakos (1998), Burton and Mooney (1998)
individuals is well supported as online and Tumolo (2001)
and e-marketplaces facilitate this ability.
For example, Ford anticipates that it will
be able to supply car dealerships with
special order models within 2 weeks
through receiving online specifications
Many marketplaces support instant Baron et al. (2000), Stanoevska-Slabeva
updates of catalogues and price lists, and Schmid (2000) and Tumolo (2001)
product specifications and
configurations. Traditional catalogues are
expensive to print and distribute and
require additional printing costs to
update them. Cost of online updates are
substantially lower
(continued)
306
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Table I
Advantage
Lower transaction costs
Differentiation of products and
services/customisation
Ability to enter supply chain for
larger companies
the difficulties of smaller companies and their
differing needs (Howarth, 2002; Stockdale and
Standing, 2003b). Although some marketplaces
carry statements that they support smaller
businesses, they may charge initial fees that are
beyond the resources of many or require an
understanding of, and commitment to, specialist
software. SMEs are unlikely to commit resources
without a recognisable return of benefits for the
investment of time and money (Korchak and
Rodman, 2001).
Lack of standards
The lack of a common framework for buyers and
sellers hinders the development of many
marketplaces. Currently, e-marketplaces often
adopt their own platforms without regard to any
industry or technological standards.
Gulledge (2002) reports that over 120 standards
that extend XML have been identified. Such
variety can deter participation by large and small
firms, unwilling to commit to software and training
before they can identify returns on their
investment (Howarth, 2002; Lucking-Reiley and
Spulber, 2001).
Understanding of the environment
SMEs often do not have an understanding of the
nature of the Internet and how it interacts with
other methods of trading (Stockdale and Standing,
2003b), although this is not confined to smaller
businesses (Porter, 2001). It is important for
SMEs to understand that the Internet is not
a substitute for established methods of trading
except for companies that are created specifically
for the Internet environment such as Amazon.
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
Benefits Source
Search costs for new buyers/suppliers are Bakos (1998), Clemons et al. (1993),
substantially lower. Additionally, Malone et al. (1987), Modahl (2000) and
electronic transaction processing such as Tumolo (2001)
order entries, online payment options
and order tracking are seen as more
efficient and less expensive
The transparency of information in Brunn et al. (2002), Burton and Mooney
e-marketplaces enables companies to (1998) and Korchak and Rodman (2001)
identify where they can differentiate their
products and services from competing
companies within the same marketplace
Large companies have broadened their Erbschloe (1999) and
supplier bases through the use of Korchak and Rodman (2001)
e-marketplaces. The advantages of cost
and speed that can be gained from
trading on online are available to
companies of all sizes and reduce the
barriers that have hindered smaller
companies attempting to enter the
supply chains of larger companies
For the majority of business the Internet is a
complementary tool that can enhance their current
business.
Supply chain integration
Smaller companies do not often see themselves as
part of a large supply chain. They underestimate
how e-commerce can facilitate interaction with
larger firms within a supply chain by enabling the
sharing of information, electronic ordering
(thereby dispensing with a paper system),
electronic fulfilment, tracking, and efficiencies in
cost and time (Korchak and Rodman, 2001).
If SMEs do not understand that e-commerce
competencies will support their ability to function
within the larger supply chain they will lose out to
other firms which can operate in the electronic
market.
Industry environment
Many SMEs operate within a relationship
environment that does not encourage innovation
and there is little incentive to be the first mover in
the transition to e-marketplaces. For example,
while small companies that supply Ford or Renault
must go to the major automobile marketplace,
Covisint, to maintain that relationship, there is
little incentive for publishers to go online as few of
their buyers (bookshops) are sophisticated
e-commerce users.
Identification of benefits
The perceived instability of the electronic
environment hinders the progress of e-marketplace
adoption. SMEs rarely benefit from being first
movers unless action is in response to innovations
307
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
in their external networks (North and Smallbone,
2000) and therefore, there is no incentive to
undertake risk. There has to be some realistic
immediate benefits to encourage the first move,
before longer-term benefits become an issue
(Korchak and Rodman, 2001).
Global trading
The ability to trade globally is often associated
with the electronic environment. While
e-marketplaces can support many of the processes
required to achieve global purchases and sales
through offering customs advice, currency
exchange and shipping services, many pitfalls
remain (Ives and Jarvenpaa, 1991; Peppard,
1999). These can include language difficulties,
cultural differences and import/export legislation.
These are not insurmountable obstacles, but
require recognition and understanding.
Financial constraints
There may be an adverse effect on credit lines from
trading through e-marketplaces for SMEs that do
not have the financial backup to appreciate any
differences in the trading environment. There is
some anecdotal evidence that SME credit lines are
not geared to frequently changing buyer/supplier
relationships and financial institutions may be
wary of extending credit for Internet-based
trading.
There is a wide variety of potential benefits and
barriers to e-marketplace participation for small
and medium sized businesses and there is no easy
recipe for overcoming the challenges and realising
the benefits. Some barriers relate to recognised
problems common to SME e-commerce adoption,
such as connectivity, while others are more specific
to the individual company such as lack of
resources. In contrast, the realisation of the
benefits of participation generally rests with the
ability of individual SMEs to identify opportunities
and to plan their online trading effectively within
the constraints of their industry environment.
Effective planning of a participation strategy is
therefore of critical importance if an SME is
to realise benefits.
Strategies for e-marketplace
participation
SMEs’ participation in e-commerce is not often
characterised by formal strategic planning (Hall,
1995), although by using a broader definition of
strategy, Chau (2003) found wide evidence of
some element of strategic planning in SMEs in
Australia. His definition included businesses that
have made significant alterations to business
308
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
processes to incorporate e-commerce components.
Chau (2003) further found that business planning
and strategic goals enable SMEs to establish their
businesses at a desired level of participation,
thereby by-passing the stages of growth models
(McKay et al., 2000; Poon and Swatman, 1997)
familiar to researchers in this area.
There are different motivations for SMEs to
move into online trading through e-marketplaces
and this may have an influence on the level of
strategic planning achieved by a firm. It is only
recently that motivations for e-commerce adoption
in smaller businesses have been addressed
(Levy and Powell, 2003; Mehrtens et al., 2001).
Stockdale and Standing (2003b) identified two
groups of SMEs that have overcome the barriers to
e-marketplace participation. The first group are
recognisable in Chau’s (2003) description of firms
that have made some changes to business practices
to benefit from a recognised level of e-commerce
participation. These firms identify a need to
participate either because their industry sector is
active in online or because they recognise an
opportunity to grow their business through
e-marketplace activity. The second group are those
SMEs that follow trading partners online in order
to retain a relationship. Where the larger partner
has moved to an e-marketplace, smaller trading
partners are forced into the electronic
environment, often at a sophisticated level.
Strategic planning in these cases is less evident.
The eight barriers to e-marketplace
participation identified in this paper fall into two
groups (Figure 5). Three of the barriers are
generally specific and not within the capability of
individual firms to overcome, although knowledge
of them will contribute to a more effective strategy.
These are lack of understanding of SMEs’ needs,
the lack of a common technological standard
and the level of e-competencies within industry
sectors. Nevertheless, an understanding of these
barriers will enable SMEs to more effectively select
an e-marketplace to participate by researching the
way a prospective marketplace supports individual
firms, the technology required to participate and
the types of marketplace servicing a particular
industry. An SME in an industry sector that is slow
in e-commerce uptake may still choose to seek out
a marketplace that will enable it to source goods or
extend its market. Conversely, a smaller firm in a
technologically advanced industry sector will need
to research the different types of marketplace
available and have an understanding of what each
offers.
The remaining five barriers constitute elements
that require a measure of strategic planning if
participation is to bring benefits to the firm.
For example, SMEs require an understanding of
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Figure 5 SME barriers to e-marketplace participation
the environment in which they intend to operate.
This includes consideration of: where use of the
e-marketplace sits within the business processes of
the firm and how use of it can contribute. Use of an
e-marketplace in itself will not enable realisation of
benefits if there is no understanding of the
environment and what benefits are available.
This is particularly true of SMEs that have been
forced into a marketplace to retain a trading
partnership. With an understanding of the
environment and the potential benefits to be
gained, the e-competencies forced on them can
be used to further explore the marketplace and
gain the advantages of new partners and markets.
In essence, if the SME is proactive rather than
reactive in the enforced situation then much can
be gained from the skills they have learned.
These firms are often within the supply chains of
large organisations, but may fail to recognise that,
with e-competencies, they are attractive to other
organisations seeking to broaden their supplier
bases. SMEs moving online to enhance their
businesses can also benefit from studying the
supply chains of large organisations, many of
which are e-enabled. There may be financial
constraints on smaller firms extending their
business to non-traditional markets and
consideration of credit lines, payments and even
currencies is a vital part of the planning process.
Electronic marketplaces that support financial
services can be of great value to smaller firms that
have limited resources. However, technology
advances in e-money and e-payment systems are
predicted to overcome many of the financial
constraints facing smaller firms (Fariselli et al.,
1999).
For SMEs seeking to move into global markets,
seen as a major benefit of electronic marketplaces,
careful selection of an e-marketplace that can
facilitate payment, logistics and other such services
309
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
can be a major advantage. An understanding of the
complex social, cultural and economic issues and
the advantages to be gained will contribute to the
planning process and the realisation of benefits.
The perception that operating in a global market is
an extension of operating within regional
boundaries may hold true in the short-term, but
sustained benefit realisation requires greater
understanding (Peppard, 1999). This will be
particularly true for SMEs with their limited
resources, although the flexibility and innovation
that characterises many SMEs (North and
Smallbone, 2000) can be of particular advantage
in the global market.
Conclusions
Although many SMEs are struggling with
e-commerce adoption, understanding of the
e-marketplace environment is important.
The opportunities to extend trading beyond
traditional market boundaries make all firms
vulnerable to new sources of competition. Many
smaller firms that supply large organisations are
being forced into the e-environment to retain their
trading partners, while other firms seek to extend
their markets or to enhance their businesses by
trading beyond their traditional customer bases.
A recognition of both benefits and barriers that
firms face in entering the e-marketplace
environment will enable SMEs to more effectively
plan their participation and realise the benefits of
e-marketplace trading. Where firms follow an
established partner online, they can gain further
benefits and seek new relationships if they
understand the environment in which they find
themselves and use their e-competencies to
actively participate in what the e-marketplace has
to offer.
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Implications for future research
It is beyond the scope of this paper to determine
the reasons for low e-commerce adoption levels,
but they are a serious hindrance to the use of
e-marketplaces and challenge the ability of SMEs
to compete effectively in a global market. The
literature shows that there is good understanding,
at least at academic and government levels, of the
benefits and barriers to e-commerce. Therefore,
empirical research into SMEs’ motivations to
adopt and use e-commerce will enhance initiatives
to encourage participation in e-marketplaces.
An understanding of the benefits and barriers of
e-marketplaces is the first step for SMEs in the
move to online trading. There is a significant need
for smaller firms to trade online either to follow
existing partners, compete effectively for their
traditional markets or to enter new markets.
Although research into e-marketplaces is
underway it is mainly confined to the activities of
large organisations and the impact on SMEs has
yet to be addressed. Empirical evidence is required
to determine why SMEs trade through
e-marketplaces, and their levels of use and
competency in doing so. Studies of the threats to
SMEs in regional areas from outside competition
and the ability of SMEs to extend their markets are
needed to inform frameworks to encourage more
participation.
References
ABS (2003), Business Use of Information Technology, Bureau of
Statistics, Canberra.
van Akkeren, J. and Cavaye, A.L.M. (1999), “Factors affecting
entry-level Internet technology adoption by small business
in Australia – evidence from three cases”, Journal of
Systems and Information Technology, Vol. 3 No. 2,
pp. 33-48.
Bakos, J.Y. (1991), “A strategic analysis of electronic
marketplaces”, MIS Quarterly, Vol. 15 No. 3, pp. 295-310.
Bakos, J.Y. (1998), “The emerging role of electronic marketplaces
on the Internet”, Communications of the ACM, Vol. 41
No. 8, pp. 35-42.
Bar, F. (2002), “The construction of marketplace architecture”,
available at: www.standford.edu/,fbar/Publications/
Brookings-e-marketplace.pdf, (accessed 23 May 2002)
Baron, J.P., Shaw, M.J. and Bailey, A.D. (2000), “Web-based
e-catolog systems in B2B procurement”, Communications
of the ACM, Vol. 43 No. 5, pp. 93-100.
Bode, S. and Burn, J.M. (2002), “Strategies for consultancy
engagement for e-business development – a case analysis
of Australian SMEs”, in Burgess, S. (Ed.), Managing
Information Technology in Small Business: Challenges and
Solutions, Ideas Group, Melbourne.
Brunn, P., Jensen, M. and Skovgaard, J. (2002), “E-marketplaces:
crafting a winning strategy”, European Management
Journal, Vol. 20 No. 3, pp. 286-98.
Burton, A.F. and Mooney, J.G. (1998), “The evolution of
electronic marketplaces: an exploratory study of
310
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
Internet-based electronic commerce within the American
independent insurance agency system”, Australian Journal
of Information Systems, (Special ed.), pp. 20-42.
Chau, S. (2003), “The use of e-commerce amongst thirty-four
Australian SMEs: an experiment or a strategic business
tool?”, Journal of Systems and Information Technology,
Vol. 7 No. 1, pp. 49-66.
Choudhury, V., Hartzel, K.S. and Konsynski, B.R. (1998), “Uses
and consequences of electronic markets: an empirical
investigation in the aircraft parts industry”, MIS Quarterly,
Vol. 22 No. 4, pp. 471-507.
Clemons, E.K., Reddi, S.P. and Row, M.C. (1993), “The impact of
information technology on the organization of economic
activity: the ‘move to the middle’ hypothesis”, Journal of
Management Information Systems, Vol. 10 No. 2.
Cragg, P.B. and King, M. (1993), “Small-firm computing:
motivators and inhibitors”, MIS Quarterly, Vol. 17 No. 1,
pp. 47-60.
Darch, H. and Lucas, T. (2002), “Training as an e-commerce
enabler”, Journal of Workplace Learning, Vol. 14 No. 4,
pp. 148-56.
Davis, B. (2001), “Under the hammer”, Professional Engineering,
Vol. 14 No. 3, pp. 45-6.
Deeter-Schmelz, D.R., Bizzari, A., Graham, R. and Howdyshell, C.
(2001), “Business-to-business online purchasing:
suppliers’ impact on buyers’ adoption and usage intent”,
Journal of Supply Chain Management, Vol. 37 No. 1,
pp. 4-10.
Downes, L. and Mui, C. (1998), Unleashing the Killer App: Digital
Strategies for Market Dominance, Harvard Business School
Press, Boston, MA.
Duan, Y., Mullins, R., Hamblin, D. and Stanek, S. (2002),
“Addressing ICTs skill challenges in SMEs: insights from
three country investigations”, Journal of European
Industrial Training, Vol. 26 Nos 8/9, pp. 430-45.
Erbschloe, M. (1999), “Working with SMEs in the supply chain”,
Digital Systems Report, Vol. 21 No. 4, pp. 22-3.
Essig, M. and Arnold, U. (2001), “Electronic procurement in
supply chain management: an information economics-
based analysis of electronic markets”, Journal of Supply
Chain Management, Vol. 37 No. 4, pp. 43-9.
Fariselli, P., Oughton, C., Picory, C. and Sugden, R. (1999),
“Electronic commerce and the future for SMEs in a global
market place: networking and public policies”, Small
Business Economics, Vol. 12 No. 3, pp. 261-76.
Federal Trade Commission (2000), “Entering the 21st century:
competition policy in the world of B2B electronic
marketplaces”, The Federal Trade Commission B2B Public
Workshop.
Forrester Research (2000), “The e-marketplace shakeout”,
available at: www.forrester.com/ER/Baseline, (accessed
19 April 2001).
Goode, S. (2002), “Management attitudes toward the World
Wide Web in Australian small business”, Information
Systems Management, pp. 45-8.
Grieger, M. (2003), “Electronic marketplaces: a literature review
and a call for supply chain management research”,
European Journal of Operational Research, Vol. 144,
pp. 280-94.
Gulledge, T. (2002), “B2B e-marketplaces and small and medium
sized enterprises”, Computers in Industry, Vol. 49 No. 1,
pp. 47-58.
Hagel, J. and Armstrong, A.G. (1997), Net Gain: Expanding
Markets Through Virtual Communities, Harvard Business
School Press, Boston, MA.
Hall, G. (1995), Surviving and Prospering in the Small Firm Sector,
Routledge, London.
 Benefits and barriers of electronic marketplace participation
Rosemary Stockdale and Craig Standing
Hermanek, M., Schlemmer, C., Hope, B. and Huff, S.L. (2001),
“Critical success factors in business-to-business
e-commerce: the view of IS managers”, paper presented
at the PACIS, Seoul.
Howarth, B. (2002), “E-survivors: get ‘e’ or get out”, available at:
www.aconex.com.au/Aconex/news/brw_ecommerce,
(accessed 28 October 2002)
Hurwitz (2000), “E-marketplaces: Issues, risks, and requirements
for success”, White Paper, Hurwitz Group Inc.,
Framingham, MA.
Ives, B. and Jarvenpaa, S.L. (1991), “Applications of global
information technology: key issues for management”,
MIS Quarterly, Vol. 15 No. 1.
Jutla, D., Bodorik, P. and Dhaliqal, J. (2002), “Supporting the
e-business readiness of small and medium-sized
enterprises: approaches and metrics”, Internet Research:
Electronic Networking Applications and Policy, Vol. 12
No. 2, pp. 139-64.
Kambil, A. and van Heck, E. (2002), Making Markets. How Firms
can Design and Profit From Online Auctions and
Exchanges, Harvard Business School Press, Boston, MA.
Kaplan, S. and Sawhney, M. (2000), “E-hubs: the new B2B
marketplaces”, Harvard Business Review.
Korchak, R. and Rodman, R. (2001), “E-business adoption
among US small manufacturers and the role of
manufacturing extension”, Economic Development
Review, Vol. 17 No. 3, pp. 20-5.
Levy, M. and Powell, P. (2003), “Exploring SME Internet
adoption: towards a contingent model”, Electronic
Markets, Vol. 13 No. 2, pp. 173-81.
Lewis, R. and Cockrill, A. (2002), “Going global – remaining
local: the impact of e-commerce on small retail firms in
Wales”, International Journal of Information
Management, Vol. 22, pp. 195-209.
Lin, B. and Hsieh, C-T. (2000), “Online procurement:
implementation and managerial implications”, Human
Systems Management, Vol. 19 No. 2, pp. 105-10.
Lucking-Reiley, D. and Spulber, D.F. (2001), “Business-to-
business electronic commerce”, Journal of Economic
Perspectives, Vol. 15 No. 1, pp. 55-68.
McKay, J., Prananto, A. and Marshall, P. (2000), “E-business
maturity, the SOG-e model”, paper presented at the 11th
Australasian Conference on Information Systems,
December, Brisbane.
McMillan, J. (2002), Reinventing the Bazaar. A Natural History of
Markets, W. W. Norton and Company Inc., New York, NY.
Malone, T.W., Yates, J. and Benjamin, R.I. (1987), “Electronic
markets and electronic hierarchies”, Communications of
the ACM, Vol. 30 No. 6, pp. 484-97.
Mehrtens, J., Cragg, P.B. and Mills, A. (2001), “A model of
Internet adoption by SMEs”, Information and
Management, Vol. 39, pp. 165-76.
Modahl, M. (2000), Now or Never. How Companies Must
Change Today to Win the Battle for the Internet Consumer,
Orion, London.
North, D. and Smallbone, D. (2000), “Innovative activity in SMEs
and rural economic development: some evidence from
England”, European Planning Studies, Vol. 8 No. 1,
pp. 87-106.
Peppard, J. (1999), “Information management in the global
enterprise: an organising framework”, European Journal
of Information Systems, Vol. 8, pp. 77-94.
311
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 301–311
Poon, S. (2000), “Business environment and Internet commerce
benefit – a small business perspective”, European Journal
of Information Systems, Vol. 9, pp. 72-81.
Poon, S. and Swatman, P. (1997), “Determinants of small
business Internet usage: a multimedia investigation of
perceived benefits”, paper presented at the 10th
International Bled Electronic Commerce Conference, Bled,
June.
Poon, S. and Swatman, P. (1999), “An exploratory study of small
business Internet commerce issues”, Information and
Management, Vol. 35, pp. 9-18.
Porter, M.E. (2001), “Strategy and the Internet”, Harvard
Business Review, pp. 63-78.
Raisch, W.D. (2001), The e-marketplace. Strategies for Success in
B2B ecommerce, McGraw-Hill, New York, NY.
Sculley, A.B. and Woods, W.A. (2001), B2B Exchanges. The Killer
Application in the Business-to-Business Internet
Revolution, Harper & Collins, New York, NY.
Senn, J.A. (2000), “Business to business e-commerce”,
Information Systems Management, pp. 23-32.
Skjott-Larsen, T., Kotzab, H. and Grieger, M. (2003),
“Electronic marketplaces and supply chain
relationships”, Industrial Marketing Management, Vol. 32,
pp. 199-210.
Standing, C., Sims, I., Stockdale, R. and Wassenaar, A. (2003),
“Can e-marketplaces bridge the digital divide?”, paper
presented at the Organizational Information Systems in
the Context of Globalization. IFIP TC8 and TC9 Working
Conference, Athens.
Stanoevska-Slabeva, K. and Schmid, B. (2000), “Internet
electronic product catalogs: an approach beyond simple
keywords and multimedia”, Computer Networks, Vol. 32,
pp. 701-15.
Stockdale, R. and Standing, C. (2003a), “The effect of B2B online
reverse auctions on buyer-supplier relationships”, paper
presented at the 14th Australian Conference on
Information Systems, Perth.
Stockdale, R. and Standing, C. (2003b), “Increasing the adoption
of electronic commerce by SMEs: reflections from an
empirical study”, paper presented at the 4th International
We-B Conference, Perth, 24-25 November.
Timmers, P. (1999), Electronic Commerce, Wiley, Chichester.
Tumolo, M. (2001), “Business-to-business exchanges”,
Information Systems Management, pp. 54-62.
Van Beveren, J. and Thomson, H. (2002), “The use of electronic
commerce by SMEs in Victoria, Australia”, Journal of Small
Business Management, Vol. 40 No. 3, pp. 250-3.
Walczuch, R., Van Braven, G. and Lundgren, H. (2000),
“Internet adoption barriers for small firms in
The Netherlands”, European Management Journal, Vol. 18
No. 5, pp. 561-72.
Weill, P. and Vitale, M. (2001), Place to Space. Migrating to
e-business Models, Harvard Business School Press,
Boston, MA.
Further reading
Standing, C. and Stockdale, R. (2003), Electronic Marketplaces
and SMEs, Rural Industries Research and Development
Corporation, Perth.

Other papers
Information technology
evaluation: classifying
indirect costs using the
structured case method
Peter E.D. Love
Ahmad Ghoneim and
Zahir Irani
The authors
Peter E.D. Love is based at We-B Centre, School of
Management Information Systems, Edith Cowan University,
Joondalup, Australia.
Ahmad Ghoneim and Zahir Irani are both based at the
Department of Information Systems and Computing Brunel
University, Middlesex, United Kingdom.
Keywords
Electronic commerce, Competitive advantage, Costs,
Research methods
Abstract
Many companies are increasing their expenditure on information
technology and information systems (IT/IS) to obtain or even
sustain a competitive advantage in their respective
marketplaces. Many managers, however, are often left with the
quandary of how to evaluate their investments in technologies.
Reasons of this difficulty have been suggested in the normative
literature as centring on the socio-technical (human,
organisational and technical) dimensions associated with the
adoption of IT/IS. The inability of managers to determine the
true costs of deploying IT/IS is considered attributable to a
lack of knowledge and understanding of IT/IS related costs.
In developing a broader picture of such cost dimensions and
their respective taxonomies, the research presented in this paper
uses a structured case method to gain an understanding of how
a construction firm embraced the IT evaluation process. A review
of the IT cost literature is presented and a conceptual IT
evaluation framework that focuses on indirect costs is proposed.
Electronic access
The Emerald Research Register for this journal is
available at
www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is
available at
www.emeraldinsight.com/1741-0398.htm
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · pp. 312–325
q Emerald Group Publishing Limited · ISSN 1741-0398
DOI 10.1108/17410390410548724
312
Introduction
The problems of evaluating and justifying
investments in information technology and
information systems (IT/IS) are not new.
Information systems have always been taking too
long to develop, cost too much to implement and
maintain, and are frequently not perceived to be
delivering the business benefits which were initially
intended. In recent years, however, the changing
role of IS in organisations has given new impetus to
the problem of IT/IS evaluation. The high
expenditure on IT/IS, the growing usage that
penetrates to the core of organisational
functioning, together with disappointed
expectations about IT/IS impact, have all served to
raise the profile of how IT/IS investments can be
evaluated. Over a decade back, Willcocks (1994)
stated that “IS evaluation is not only an
underdeveloped, but also an undermanaged
area which organisations can increasingly ill-afford
to neglect”. This observation is still pertinent
today, perhaps even more so due to the
proliferation of information and communication
technology innovations that have penetrated the
marketplace.
The increased complexity of IT/IS combined
with the uncertainty and unpredictability
associated with its benefits point to the need for
improved IS evaluation. According to David
(1990), there is often a time lag from the adoption
of IT and when the expected benefits are achieved.
In fact, David’s research has shown that
productivity benefits begin to emerge within an
industry sector once the diffusion rate of
technology surpasses 50 per cent. Brynjolfsson and
Hitt (1996) and Stirroh (2001) have shown that it
takes time for the productivity benefits associated
with the adoption of IT to be achieved and diffused
throughout the value chair. Stirroh (2001) noted
that the agriculture and construction sectors have
been the only sectors in the US that have not
experienced increases in productivity between
1995 and 1999. These industries have lagged other
industries in embracing IT and consequently have
not been able to acquire the benefits of increased
productivity. Like the US, the construction sector
in Australia is “laggard” in terms of its productivity
output and adoption of IT, when compared to
other sectors of the economy (DIST, 1998; Yellow
Pages, 2002).
The construction industry, through the
products that it creates, its size and ability to create
employment, has the potential to influence an
The authors acknowledge the financial support
provided by the Australian Research Council and
Engineering and Physical Sciences Research Council
(GR/M95066 and GR/R08025).
 Information technology evaluation: classifying indirect costs The Journal of Enterprise Information Management
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani Volume 17 · Number 4 · 2004 · 312–325

economy’s gross domestic product (GDP) more Figure 1 IT investment stakeholders

than any other service industry. Stoeckerl
and Quirke (1992) explain that an increase
of 10 per cent in the Australian construction
industry’s efficiency, would improve the
economy’s service industry contribution to GDP
by over 2.5 per cent. Consequently, it is essential
that the industry operate in an efficient and
productive way. The Latham (1994) Report,
which investigated ways of improving the UK’s
construction industry, suggested that the effective
implementation of IT could reduce project costs
by as much as 30 per cent. The industry’s
conservative nature and the short-term focus of
organisations (i.e. a focus on winning the next
project) has tended to inhibit investments in IT,
even though there have been calls from the
Australian Federal Government to increase IT
adoption rates and investments in supporting
infrastructure (DIST, 1998). While many
construction organisations are readily aware of the
strategic benefits that can be leveraged from ITand
e-commerce applications, many are reluctant to
invest in such technologies. The reasons for this
centre around the hidden costs involved with expectations from the IT/IS investment. This
implementation (Ghoneim et al., 2003a, b; concept demonstrates that any form of evaluation
Mohamed and Irani, 2002). There is a need to must be multifaceted and seek to have their
identify the indirect costs of developing an IT/IS inclusion whilst considering the agendas of the
infrastructure, so that such cost factors can be different stakeholders involved.
included in the investment justification process. .
Organization (Management) – are interested in
In doing so, this can enable key decision-makers the gains [financial / and other] generated by
to manage the potential risk of cost increases that the investment. They seek to ensure that the
may occur throughout an IT/IS projects life cycle. project is implemented on time, within budget
Against the contextual backdrop a conceptual IT and according to user requirements.
evaluation framework for classifying the indirect .
Users – the technology should meet their
costs of an IS is examined using a structured case requirements while integrating flexibility to
method, with reference to an organisation from the adapt to the changing requirements of users/
Australian construction industry. customers.
.
Project team (Implementers) – focus on short-
term criteria set by sponsors (used to judge
their performance).
IT investments stakeholders .
Supporters (Subcontractors) – focus on short-
term criteria
Information system projects are increasingly a .
Stakeholders (Non-benefiting, non-influencing)
multi-stakeholder investment that impacts the – consists of many groups, each with its
organisation and increasingly its supply or service own goals and objectives. They might
chain. According to Milis and Mercken (2004), support or oppose the investment, which
there are five parties involved in information might possibly be in the form of covert
technology investments; each having their own set resistance.
of objectives and expectations. If the evaluation of
IT/IS investments is only financially based, and
centre on the use of traditional appraisal IT/IS evaluation: a focus on indirect costs
techniques then, the process only serves the
objective of management. This means neglecting The implications of ignoring “indirect” costs can
all other parties’ objectives and accordingly, failing have far-reaching consequences for companies.
to incorporate crucial factors that might affect the For example, reduced productivity may result
willingness of these parties to cooperate in realising from employees who have not been sufficiently
the objectives of the investment. Figure 1 shows trained, and loss of experienced employees to
the different parties (stakeholders) and their competitors. Li et al. (2000) suggests that firms
313
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
may only realise the significance of these cost
factors once they start to develop an IT
infrastructure. Fundamentally, poor IT
decision-making can result in financial losses,
which can translate into a loss in competitiveness,
and even jobs. The costs associated with such
losses are invariably passed on to the customer.
Research undertaken by Dier and Mooney
(1994) revealed that on-going costs
(i.e. maintaining and modifying an application) are
incurred from year to year and can account for as
much as 80 per cent of the lifetime cost of in-house
developed systems and 91 per cent of the lifetime
cost of software packages systems. Dier and
Mooney (1994) also found that managers tended
to overestimate the initial costs (i.e. hardware,
software and systems developments costs) and
underestimate ongoing costs and as a result lead to
a huge gap between the estimated allocated budget
and the actual cost experienced by organizations.
This could be attributed to the fact that managers
regard the investment justification process as a
hurdle that needs to overcome, and not as
technique for evaluating the worth of
implementing IT. Ghoneim et al. (2003c) argued
that a major difficulty associated with IT costing
related to identification costs, particularly those of
an indirect nature. According to Hochstrasser
(1992), indirect costs can be four times greater
than those of a direct nature. Whilst some
managers may be unaware of indirect costs, others
simply choose to ignore them to gain senior
management’s support by trying to minimize this
cost portfolio and at the same time increasing the
benefits portfolio (Alshawi et al., 2000).
Indirect costs are typically comprised of human
and organisational factors. One of the largest
“indirect” human cost being that of management
time (Irani et al., 1999). In other words, that time
was specifically spent on integrating new systems
into current work practices. In the case of newly
adopted technologies, management may spend
time revising, approving and subsequently,
amending their IT related strategies.
A significant amount of resource will also be
used to investigate the potential of the IT, and in
experimenting with new information flows and
modified reporting structures (Love and
Gunasekaran, 1997; Love et al., 1998). Wheatley
(1997) suggests that a further “indirect” human
cost, which is often overlooked, is that of system
support and trouble-shooting. Wheatley (1997)
revealed that many organisations could not
estimate the additional cost of supporting IT in
relation to its original purchase price. According to
Wheatley (1997), typical lifetime support costs are
at least 400 per cent of the original purchase price.
Further “indirect” costs may result from
314
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
employees developing new skills, and therefore,
increasing their flexibility/overall contribution
towards the organisation. These employees may
then request revised pay scales and benefit
packages, which the organisation may have to offer
to retain expertise. Clearly, such “indirect” costs
associated with employee pay and rewards,
together with the cost implications of increases in
staff turnover need capturing, and bringing into
the IT evaluation arena.
Mohamed and Irani (2002) proposed a two-tier
system for classifying indirect human costs. The
first tier refers to management, employee, financial
and maintenance, with a second tier to particular
cost components. In the first tier, management
decides on the amount of IT expenditure;
employees refers to all people related issues within
the organisation; financial refers to the allocation
of budgets; and maintenance to the development
and implementation of technology. The second
tier identifies indirect cost components with a
number of cost classifications, which are as
follows.
.
Time. Management time is an indirect cost in
terms of the transfer of skills through training
among staff, as neither staff nor management
will be doing their routine jobs during that
period. Bannister (1999) supports this view
by stating that the introduction of a new
system will require training. Unless the
organisation obtains a time recording
system, the indirect cost of the time spent
in training would not usually show up as
an IS cost, but identified as a hidden or
personnel related cost. Such costs include
the time spent reading manuals, self-help
activities, and informal job training, which are
untraceable.
.
Learning costs. As new users or new systems
are introduced to the organisation, users are
likely to embrace a learning curve. As a result,
a temporary loss in productivity may be
experienced, as the user becomes familiar with
the new system. There is also the possibility
that mistakes will occur during the
implementation, migration or running of the
system. It takes time to rectify problems as
well as become familiar with the peculiarities
of a system.
.
Costs of resistance. The introduction of a new
IS could result in an unexpected political
power shift that can result in some individuals
or groups resisting the change initiative.
As a result, staff engage in other staff
interdependently uses behaviour that
endeavours to result in disruption or removal
of an entire system. This is an influential
indirect human cost, as it has a great impact
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
on the social fabric of the organisation. Not
only does management have to rectify the
consequences of resistance but also bear the
cost of resolving the actual resistance.
Moreover, lack of commitment to change
could result in a non-operational working
environment, user resistance resulting in
escalating operational costs.
.
Effort and dedication. This occurs when a new
system is introduced, as management at all
levels spend time exploring its boundaries,
discovering the business potential, absorbing
the transition from the old system to the new
system, as well as identifying integration
issues. Here managers are being less
productive and thus not adding transparent
value to the organisation. The time utilised by
management and staff incorporating the new
system into the existing organisational setting
results in an indirect cost. Those members of
staff influenced by the new system will take
time to become familiar with it and as a result
this hinders their daily operations.
.
Cost of redefining roles. The introduction of IS
in some cases can lead to organisational
restructuring. The dismantling of an
organisations’ hierarchy and reduction in the
number of managerial levels, can lead to
middle management being somewhat exposed
and therefore, lead to a change in roles and
responsibilities. Kaye et al. (1995) suggests
that it may also lead to the introduction of
training, redundancy, or perhaps promotion.
. Missed-costs. Bannister (1999) states that one
of the major problems in tracing IS costs is
mis-assigning them. Displacement costs (also
called re-allocation) are proposed as being one
of these mis-assigned costs, and are explained
through the following example. If a member of
the IS staff goes on training abroad, then cost
of the aircraft is recorded as a travel expense,
the accommodation as miscellaneous and
other expenses as entertainment, when
essentially they are all actually IS training
costs. Bannister (1999) refers to disruption
and displacement costs as missed costs.
.
Reduction in knowledge base. This is usually a
result of high staff turnover, or redundancies,
in the organisation. Some organisations may
reduce labour costs, believing that they are
justifying their investment in IT. Nevertheless,
this is impossible to measure in financial
terms, and can result in a significant change in
the knowledge base of the organisation.
.
Moral hazard. Mende et al. (1994) refers to
this as “professionalism”. It is defined to be
the state in which the IS managers are
interested in gaining knowledge that will help
315
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
them to determine their job market value
rather than being interested in organisational
benefits. It could occur when decision rights
are appointed to individuals that are expert.
Managers may use their decision right to
maintain their own interest rather than trying
to meet organisational objectives (Dirks and
Van Lent, 1997; Irani and Love, 2001).
. Deskilling. This is the inability to fully utilise
the potential skills of employees. For example,
as a result of a new system being introduced,
the organisation may assign high skilled staff
to undertake low skilled tasks (Kaye et al.,
1995). As a result, either the employee may
seek a different job or the organisation
continues to assign high salaries for less worth
job roles. Investing in IT at the expense of
reduction in labour cost may lead to a
situation where it is difficult to predict
accurately the costs involved, as there is less
transparency. As an outcome, significant costs
could result from losing time and money
already invested in the employee that has left
as well as requiring the same amount of money
(if not more) for recruiting a substitute
employee.
Because of the complexity associated with IS
projects, various authors have been reserved in
recommending guidelines for evaluation purposes.
Examples of recommendations that have been put
forward are summarised as follows.
.
Kaplan (1984) explained that management
should not loose sight of the short-term
financial returns from IT while being
overwhelmed with its wider long term
strategic implications.
.
Farbey et al. (1993) stated that intangible
benefits are valuable organizational assets that
cannot be quantified in monitory term.
. Renkema and Berghout (1997) identified a
plethora of investment appraisal techniques
for the appraisal process, but there is no
consensus on a single best approach.
.
Weill and Broadbent (1998) stated that IS
should be seen as an integral part of an
organizational infrastructure.
.
Hides et al. (2000) described the difficulties
in maximizing the portfolio of IS benefits
and that minimizing the cost of IS is a
major constraint on organizational
IT investment.
.
Irani and Love (2001) presented empirical
evidence that managers are increasingly aware
that investment related costs occur during the
systems life cycle and as such, they find it
difficult to identify, manage and control IT/IS
related costs.
 Information technology evaluation: classifying indirect costs The Journal of Enterprise Information Management
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani Volume 17 · Number 4 · 2004 · 312–325

Cost taxonomies According to Table II, there is no single cost

Table I gives a summary of the limited research
surrounding cost taxonomies. Notably, there has
been limited identification of associated costs
elements within the taxonomies identified (with
the exception of Irani and Love, 2001). Thus, the
taxonomies reported in the literature are of little
use to decision-makers in their current form.
Hence, there is much need for further research to
identify cost elements and establish new and
validate current taxonomies. The identification of
such costs and appropriate grouping into
taxonomies will therefore, support improved
ex-ante investment decision-making, and the
controlling and reduction of IT/IS costs through
benchmarking during post-implementation
evaluation.
Each of the cost taxonomies encompasses a
different set of cost factors, since they all have their
different way of classifying costs. Yet, there are a
number of cost elements that are shared between
most of the eight cost elements. Table II outlines
those cost elements that are shared by two or more
cost taxonomies, and represents the cost factors
ordered descending from the most to the least cited
by authors. The table also indicates the number of
cost elements that shared the same number of
occurrences in the literature taxonomies.
factor that is identified by all eight models. This
does not necessarily imply that the least cited cost
factors are insignificant or less important. For
instance, organizational restructuring, overheads,
modifications, productivity loss, and management
time among others were the least cited. One reason
could be that academics are not aware of these
costs and/or the organizations studied for
developing the taxonomies were not familiar with
these cost factors. The three most cited cost factors
were found to be the hardware, training, and
software costs, which are easy to identify and
financially quantify. According to the literature,
these costs represent the direct initial investment
costs of IS projects.
It also noticed that the academics are
increasingly aware of the indirect costs associated
with IT/IS investments, which are cited in some
cases by five models in the case of covert resistance
and changes in salaries. Nevertheless, the majority
of the indirect costs are still relatively new to
academics, which raises doubts about the level of
awareness of practitioners and their knowledge of
such costs.
The inability of managers to identify these
various cost factors can be an indirect driver for
other costs to occur. For example, when
implementing a new system, employees acquire

Table I IT/IS cost taxonomies

Cost taxonomies Reference
Financial/Non-financial activities. These costs are classified according Kusters and Renkema (1996)
to the activities causing them thus, emphasising a causal relationship.
Hence, reactive in nature
Initial/Ongoing costs. These costs are identified and assigned during Dier and Mooney (1994)
the systems life cycle. However, they tend to be retrospective, which
make their consideration during ex-ante evaluation difficult. Yet, as
legacy systems and enterprise solutions become more integrated,
such cost taxonomies warrant closer consideration in terms of
identifying their respective cost elements
Direct/Indirect: human and organisational. The direct cost element Irani and Love (2001)
is assigned to the information technology component, whereas the
indirect elements relates to the effect of the information system
on the organisation and people
IS cost divisions – management, employee, finance, and maintenance. Mohamed and Irani (2002)
This cost taxonomy identifies a set of cost factors and sub-systems
that impact the organisation. However, this taxonomy falls short of
identifying performance measures
Initial investment/Ongoing costs. These are based around the costs Remenyi et al. (1996)
relating to the development of an information system infrastructure
(initial investment) and operation of the infrastructure (ongoing cost)
Development/Hidden costs. Costs related to the purchasing, installing, Anandarajan and Wen (1999)
training, and testing the system
Social subsystem costs. The costs that reflect the changes in the social Ryan and Harrison (2000)
subsystem bought about by a new information technology
Acquisition/Administration: control and operations costs. This taxonomy David et al. (2002)
identifies a set of cost factors that constitute the total cost of
ownership (TCO) of information technology

316
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
Table II Cost factors shared by two or more cost taxnomoies
Cost factors shared by two or more
taxonomies
2 Training
Hardware
1 Package software
3 Maintenance
Implementation risks (covert resistance)
Staff related costs (changes in salaries,
etc.)
2 Communication
Employee time
5 Development
Support
Upgrades
Installation and configuration
Business process re-engineering (BPR)
14 Operations
Systems software
Modifications
Overheads
Management/Staff resource
Management time
Cost of ownership: system support
Management effort and dedication
Productivity loss
Organizational restructuring
Infrastructure
Accommodation/travel
Displacement and disruption
Learning
new skills due to the training they underwent to
use the system effectively and efficiently. Managers
have to take into consideration that the employees
may ask for a raise in salary for improved skillful
levels. If managers are not aware of this possibility
(which in this case is an indirect cost), then they
might not be able to offer these employees their
required remuneration. Consequently, this may
have an impact on employee retention (another
indirect cost). Staff turnover will lead to
recruitment expenses and training expenses for
new staff. The point here is that not paying
attention to or trying to avoid indirect costs can
indirectly lead to a more increase in the total cost
of a project. Managers should try to identify and
plan for these costs at the beginning of the project’s
life cycle. Better management will lead to better
control of costs, which may translate into increased
profits.
Research methodology
Earlier research suggests that a firm’s “failure” with
IT is primarily attributable to not meeting user
expectations, which underlines the significance of
the “soft” human and organisational issues
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
involved in IT (Irani and Love, 2001). Bearing this
Number of in mind, the authors adopted a research
occurrences methodology that would involve and enfranchise
an organisation and their senior staff so that theory
7 and knowledge about the decision-making process
7
for IT could be derived to develop an effective
6
model for IT evaluation that considers both direct
5
and indirect benefits and costs. In this paper, we
5
explore how indirect IT cost was incurred, so we
5
can build a framework to map their occurrence.
4 Noteworthy, the framework is not developed in
4 isolation and possible benefits that can arise
3 when evaluating IT/IS have been included to
3 demonstrate how the benefits are integral to cost
3 identification. However, the case that is presented
3 focuses on the indirect cost identification of IT/IS.
3
2
2 Structured case method
2 The structured-case seeks to build theory, which
2 may be seen as “a system of interconnected ideas
2 that condense and organise knowledge” (Neuman,
2 1991, p. 30) that attempts to explain, predict and/
2 or provide understanding. The aim of adopting
2 such an approach is to discover and discuss
2 relationships between concepts, so as to build a
2 “web of meaning” with respect to the human and
2 organisational issues of IT evaluation (Carroll and
2 Swatman, 2000). The development of a series of
2 conceptual frameworks (CF) as shown in Figure 2,
2
from CF1, CF2, . . ., CFn is used to demonstrate the
process of knowledge and theory building where
CFn is the latest version of the theory built to date
(Carroll and Swatman, 2000; Carroll et al., 1998).
The theory-building process is not only inductive
but is interrelated with practice (Carroll and
Swatman, 2000; Carroll et al., 1998). Applied
research can lead to theory building, which can
lead to further field research and more theory
building. Thus, the research cycle can lead to
changes to the CF. Each new CF expresses the
pre-understanding for the next cycle, as part of
the hermeneutic circle (Gummesson, 1998).
That is, the natural human act and process of
interpretation and understanding of the world
(Carroll et al., 1998). Essentially, what is enacted is
a spiral towards understanding, as current
knowledge and theory lay the foundations for yet
another research cycle that will expand, revise or
validate the authors’ understanding of IT
evaluation in construction. This is particularly
appropriate for IT, as it is an area distinguished by
rapid changes in practice, which suggest the need
for theory, and practice to become closely
intertwined (Galliers, 1997). It is envisaged that
the structured-case will enable theory to be
developed that will reflect the concerns, problems
and issues facing construction firms (Carroll and
Swatman, 2000).
317
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
Figure 2 The conceptual framework (CF1) for IT evaluation: incorporation of indirect cost factors
Initial conceptual framework
As limited research has examined IT indirect
costs, the conceptual framework developed by
Irani et al. (2000) was adapted by including
indirect cost factors to explore those costs
experienced by a construction organisation at
an enterprise level. As mentioned above, the
authors’ particular interests centre on the
importance of the human and organisational
dimensions associated with the evaluation process,
as it is considered that the implementation of
IT can have cascading effects throughout an entire
organisation if these issues are not adequately
addressed.
Justification of CF1
The case for looking at a stakeholder perspective
during the evaluation process itself has been well
justified through the literature ( Willcocks and
Lester, 1996). The importance of establishing
318
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
taxonomies that take into consideration human
and organisational aspects is a significant
departure from conventional approaches used to
evaluate technological systems. By taking a
stakeholder perspective, it is suggested that the
process of investment and implementation of
IT should be driven by the organisations
capability, the level of competency available
and the inherent culture, values and level of
experience available within the construction firm.
It is therefore of vital importance that the following
steps are closely adhered to in preparation of
introducing change based on the implementation
of IT.
.
Corporate commitment by “selling” the need
for change and the importance of such an
investment.
.
Communicating clearly the value, benefits and
implications for such investments on people
and the organisation.
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
.
Leading the change proactively by
ensuring that the culture per se is
not averse to modernisation and investment of
IT.
.
The level of employee involvement and
participation, empowerment of employees
at various levels within the organisation.
.
A skill audit will have to take place prior to
major investments and a necessary strategy of
skill upgrade and development.
A stakeholder perspective during IT/IS investment
programs cannot and should not be considered
as a one off, project-based approach. It is
evidently clear that as construction firms enter
an era based on IT, a continuous process of
technological appraisal, evaluation, upgrading
and integration needs to be adopted. This
therefore, indicates that stakeholder orientation
should be a continuous process. Collective
experience, expertise, experimentation,
continuous improvement initiatives will all
gradually impact the benefits that can be derived
from IT adoption. The cost taxonomies proposed
in CF1 are the scope of consideration in this
research. Evaluating IT investments is no longer
appropriate to include only conventional,
economically-based costs as the justification
process needs to focus on a variety of benefits,
which may lie at a strategic, tactical and/or
operational level (Figure 2).
The research cycle
The research cycle aimed to validate or further
revise the proposed CF1. The construction firm
(ABC) used was a large construction and
engineering contractor that has constructed many
of Australia’s best known building’s and
landmarks. In 2001, the annual turnover for
ABC was approximately $430 million, and in
2002, it was approximately $475 million. ABC has
a solid client base, and it estimates that 80 per cent
of its current projects are for repeat clients.
ABC has offices throughout Australia and
South East Asia.
ABC has an extensive IT infrastructure in place.
At a project level, ABC only uses basic
technologies as devices for inter-organisational
communication. While ABC acknowledges the
importance of IT as a mechanism for improving
inter-organisational communication, the financial
rewards that may be obtained from using IT are
not considered to be beneficial in the short-term.
At an enterprise level a packaged software system
supports ABC’s management information system
(MIS). The package software system is used to
319
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
store information on a central server for
estimating, cost planning, contract administration,
and document control purposes. ABC has
upgraded and introduced new hardware and
software during the last 2 years, for example,
document imaging systems, videoconference
systems, e-mail and Internet and Intranet
systems. The direct financial cost of
implementing these technologies was estimated
to be $650,000 for the state office involved in
this study, which had an estimated turnover of
$60 million in 2002. ABC’s management team
did not undertake a rigorous IT evaluation
process and as a result, has encountered several
difficulties especially with the software system
used to support strategic the management of
business processes.
Data collection
The data collection procedure has followed the
major prescriptions by most textbooks in doing
fieldwork research (Yin, 1989). A variety of
secondary data sources were used to collect
data with regard to the development of IT
taxonomies for evaluating IT investments, such
as internal reports, budget reports, and filed
accounts. A variety of data sources have been
used to derive the findings presented in this
paper, which include interviews, observations,
illustrative materials (e.g. newsletters and other
publications that form part of the case study
organisation’s history), and past project
documentation. The authors have extensive
industrial experience in the construction industry
and have used this, together with a predefined
interview protocol to determine the data needed
for the research.
Interviews were conducted with the
regional manager, company accountant,
quality manager, estimating manager,
construction managers, project managers, and
general support staff at the enterprise-level. The
duration of each interview was approximately
30 min, where every interview was conducted
on a one-to-one basis, so as to stimulate
conversation and breakdown any barriers that
may have existed between the interviewer and
interviewee. The authors acted as a neutral
medium through which questions and answers
were transmitted and therefore, endeavoured to
eliminate bias. In considering this, bias in
interviews occurs when the interviewer tries to
adjust the wording of the question to fit the
respondent or records only selected portions of
the respondent’s answers. Most often, however,
interviewer bias results from the use of probes,
where these are follow-up questions and are
typically used by interviewers to get respondents to
 Information technology evaluation: classifying indirect costs The Journal of Enterprise Information Management
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani Volume 17 · Number 4 · 2004 · 312–325

elaborate on ambiguous or incomplete answers Table IV Classification of indirect organisational costs

(Jick, 1979; Shaughnessy and Zechmeister, 1994). Indirect organisational costs IT cost factor
In trying to clarify the respondent’s answers the
interviewer was careful not to introduce any Productivity losses Developing and adapting to new systems,
ideas that may form part of the respondent’s procedures and guidelines
Strains on resources Maximising the potential of the new technology
subsequent answer. Furthermore, the interviewer
through integrating information flows and
was also mindful of the feedback respondents
increasing information availability
gained from their verbal and non-verbal responses.
Process redesign The redesign of organisational functions,
As a result, the interviewer avoided giving
processes and reporting structures
overt signals such as smiling and nodding Organisational restructuring Covert resistance to change
approvingly when a respondent failed to answer
a question.

Decision-making
While reaping the benefits of their quality
Findings and discussions program ABC senior management (regional
manager, company accountant, and construction
The variables identified in the CF1 were found manager) considered that it was time to embrace
to be interdependent and thus, impossible to the new millennium with a revised business
them. We use the generic headings from the strategy that centred on the strategic use of IT.
proposed framework to present our initial Senior management was convinced that IT
findings and discussion. Taxonomies of would improve their organisation’s business
indirect costs that were found to be experienced performance. Attendance at business seminars
can be found in Tables II-IV. ABC considers on the strategic use of IT, as well as the
itself to be a pro-active and innovative construction encouragement of State Governments to
and engineering organisation. They have a adopt IT were factors that influenced
corporate philosophy founded on quality and managements decision to invest and develop
as result operate a continuous improvement and an extensive IT infrastructure. Senior
philosophy. The implementation of quality management considered that they were taking
assurance and subsequently CI was a long a long term view when they decided to invest
and arduous task, but ABC are now reaping in IT. The regional manager stated that
the benefits, e.g. reduced rework, repeat “. . .investing in IT will improve communication
client’s, increased market share and increased within the organisation, especially our sites.
profitability. We consider this to be a long term investment
and are committed to being an innovative
organisation”. This statement contradicts the
Table III Classification of indirect human costs claim made by Lefley (1994) that managers
Indirect human costs IT cost factor are more susceptible to make short-term
decisions with respect to new technology
Management/Staff resource Integrating computerised administration and
than any other aspect of their company.
control into work practices
Lefley (1994) further states that management
Management time Devising, approving and amending
IT/IS and marketing and procurement
is often reluctant to make long term investments,
strategies as they are more amenable to their shareholders
Cost of ownership: system Vendor support/trouble shooting costs by investing in low risk short-term projects,
support management effort and Exploring the potential of the system which show high profits. This did not appear to
dedication be the case in ABC, as the investment in IT was
Employee time Detailing, approving and amending considered to be a strategic decision that
the computerisation of estimating, would enable long term benefits to be achieved.
cost planning, planning and The company accountant stated that the
project/contract administration decision to extend and implement IT was
Employee training Being trained to manipulate considered “. . .a must, if the firm is to remain
vendor software and training others competitive in the future”. A budget was
Employee motivation Interest in computer-aided estimating prepared by the company accountant and then
and planning reduces as time passes different IT requirements were ranked in order of
Changes in salaries Pay increases based on improved preference and in terms of the greatest perceived
employee flexibility benefit by department heads (quality manager,
Staff turnover Increases in interview costs, induction costs, estimating manager, etc.). After each requirement
training costs based on the need
was ranked a cost was then allocated, albeit a
for skilled human resource
nominal figure.
320
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
Employee involvement
At no point did those people ranking the
various technologies consider the indirect
costs of implementation. Moreover, it was
revealed that end-users had not been invited to
participate in the decision-making process.
The authors’ found this surprising as ABC
had always involved their employees in the
organisation’s decision-making process. In fact,
employee involvement was seen to have been the
key to ABC’s success. So why were they not
included in deciding the IT requirements of
the firm? It was difficult to obtain an answer to
this question and is something that the authors’
found puzzling to say the least. From the
interviews undertaken with employees and
management it was simply a case “management
knew best” in this instance. It was also observed
from the interviews that certain managers
had hidden agenda’s to push, as they sought
particular technologies to improve the
effectiveness of business processes they were
involved with. Promotions and bonuses within
ABC were related to performance, so if
departments were performing well the
respective departmental manager would be
given due credit.
Leadership
There was a lack of leadership during the
decision-making and implementation process.
As a result there was a degree of confusion
within the organisation as to who was responsible
for leading the IT project. Essentially,
departmental managers were responsible for
implementing IT within their specific areas.
This created a degree of confusion as the company
accountant and quality manager, who were
responsible for implementing the packaged
computer system, did not coordinate the
installation of software to individual machines
with departmental managers. This resulted in a
delay to the systems operation, which subsequently
had negative consequences on employee morale.
Software glitches were also encountered, which in
turn had an impact on the productivity of
employees. In hindsight ABC should have
developed bespoke software that matched the
requirements and needs of end-users. It was,
however, considered cheaper (reduced
development costs) to purchase packaged
software and amend it to suit the requirements of
the firm. However, this has been a cumbersome
and time-consuming task. The costs associated
with amending the software are ongoing.
ABC acknowledges the problems associated
with purchasing packaged software and are now
considering bespoke software.
321
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
Communication
The reasons why ABC introduced new
technologies were deemed to be obvious
by management but not so to employees.
Meetings were held to discuss the implementation
process after the decision to invest had been made.
There was little communication between
management and employees, especially those on
site. One project manager made the following
comment “. . .I do realise that IT can improve
the way we do things, but what am I going to
do. I’ve never used a computer in my life.
There’s no way I can start using one now. . . I’m
almost 50”. What management had failed to do in
this instance was they did not inform staff that they
would provide training and support for those using
the new IMS.
Resource development
Time was a factor that was overlooked during
the implementation process. It was assumed that
employees would conduct their day-to-day
activities as well as become involved with the
installation of new systems and technologies.
Many employees found themselves working
overtime during the implementation phase.
Expertise was brought in from outside to
assist with the installation of software and
hardware systems. However, this was not
considered enough, as users had specific
software and hardware requirements that
had not been considered. While it was
acknowledged during the initial decision-making
process that an IT manager and a technician
were needed they were not employed until
6 months after the initial implementation phase.
This was because it was difficult to recruit
experienced IT personnel, as there was a shortage
of IT personnel with construction industry
experience.
Training costs were not considered during the
evaluation phase, they were simply overlooked.
The costs of training were found to be
considerable, as almost all people within the
organisation had to be sent on an IT training
course. Employees using the new technologies
and software programs stated that their
productivity had decreased and that they had
to work longer hours. Moreover, it was revealed
that while employees were away from their jobs
due to retraining their colleagues work patterns
were also disturbed. During this unfamiliarity
phase, the losses in productivity did not go
unnoticed by senior management. The regional
manager stated:
We made a mistake thinking that employees
would automatically take to the new technology.
Morale was poor at one stage because we failed
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
to communicate our intentions. We are now
doing our best to provide assistance and support
those employees that need it. You’ll always get
teething problems when you introduce something
new into an organisation, but sometimes you
have to take a risk and deal with the consequences
as they arise. In hindsight we should have done
things differently.
These consequences could have been accounted
for during the evaluation stage, which in turn
would have reduced the firm’s overhead and
operating costs. Noteworthy, it is estimated
that the total cost (which is ongoing) for
implementing IT may well be in excess of
$2 million, which is three times that originally
budgeted for by ABC.
The implementation of the packaged software
IMS and Intranet system meant that information
was more readily available, which improved
decision-making and communication. The
introduction of the new systems meant that single
departments were no longer needed. ABC’s
organisational structure was redesigned to
enable teams to be formed for particular projects.
The new systems required employees to become
multi-skilled, as information was readily available.
The organisational boundaries that had earlier
existed within ABC essentially acted as obstacles
to information flow and decision-making.
Business process needed to be redesigned,
as there was no further need for functional
departments. The change in organisational
structure meant that multi-skilled teams were
formed to perform activities for specific
projects. The internal layout of ABC’s
head office has become more open and over
time it is anticipated that this will influence
the culture of the firm. The redesign of the
ABC’s organisational structure resulted in
management spending time on integrating
new systems into current work practice; a cost
that had not been allowed for during the
evaluation phase.
Two estimators that were close to retirement
found the notion of using IT to perform their work
too daunting, and as a result took earlier
retirement. If staff had been involved in the
decision-making process then these employees
would have been able to prepare themselves for the
change that had been cast upon them. In this
instance, it is very difficult to demonstrate that IT
caused these employees to retire. However, the
changes imposed on the organisation would have
probably favoured newer, younger employees who
were able to learn more quickly. Thus, the older
employees may have lost their credibility because
their knowledge and experience would have been
tied to the earlier work methods. The disbenefits
often associated with IT are largely caused by the
322
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
way individuals react to change and not IT itself.
To some extent this was the case at ABC, as
employees were not confident in the decisions
made by senior management because they had not
communicated the reasons why change, in the
form of IT, was being introduced. But, however,
good a firm is, at least some employees will always
suspect the motives of management, simply
because that is how people behave when
confronted with change.
The findings presented have resulted in a
modification of the initial CF1. The revised
framework, CF2, can be seen in Figure 3. Here it
can be seen that we have separated indirect costs
into organisational and human costs and have
modified the cost factors in accordance with the
findings reported. This revised framework can now
form the basis of future research into the area of
indirect related IT costs within a holistic IT
evaluation framework. An area that also needs to
be examined in conjunction with indirect costs is
the intangible benefits associated with IT
implementation. This aspect of evaluation will be
examined in the authors’ future work with the case
study organisation.
Conclusions
Construction firms need to undertake a rigorous
evaluation process before implementing IT if they
are to achieve improvements in their business
performance, otherwise inefficiencies in decision-
making and resource deployment will prevail. In
addition, management needs to look beyond the
use of traditional modes of investment justification
(which are based on the use of economic appraisal
techniques) and consider the introduction of
indirect costs into the formal decision-making
process. This should then provide a more realistic
view of the investments’ implications rather than
providing a selective picture of quantifiable project
benefits and costs. Failure to include indirect cost
implications not only questions the value of
traditional justification processes but also has
implications regarding the actual successes of
many IT deployments. The case study identified
problems that arose from not undertaking a
rigorous investment evaluation process,
particularly in the context of indirect costs.
Problems associated with the deployment IT such
as training, lack of end-user involvement, process
redesign have been addressed by case study firm,
but at a cost! Despite the associated cost the
construction firm has experienced many benefits
such as quality of service improvements, cost
savings (reduction in clerical salaries, etc.), and
improved communication.
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
Figure 3 Revised conceptual framework (CF2) for IT evaluation: incorporation of indirect costs
References
Anandarajan, A. and Wen, H.J. (1999), “Evaluation of
information technology investment”, Management
Decision, Vol. 37 No. 4, pp. 329-37.
Alshawi, S., Irani, Z. and Jassim, A. (2000), “Novel cluster
performance matrix: a decision aid for evaluating
computer hardware”, Journal of Intelligent Systems,
Vol. 10 Nos 5/6, pp. 543-60.
Bannister, F. (1999), “What did we pay for that? The awkward
problem of IT cost”, Proceedings of the 6th European
Conference on Information Technology Evaluation, 4-5
September, Brunel University, Uxbridge.
Brynjolfsson, B.L. and Hitt, L. (1996), “Paradox lost? Firm level
evidence on the returns of IS spending”, Management
Science, Vol. 42 No. 4, pp. 541-58.
Carroll, J. and Swatman, P. (2000), “Structured-case: a
methodological framework for building theory in
information systems research”, European Journal of
Information Systems, Vol. 9 No. 4, pp. 235-42.
323
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
Carroll, J., Dawson, L.L. and Swatman, P.A. (1998), “Using
case studies to build theory: structure and rigour”,
Proceedings of 9th Australasian Conference on
Information Systems, 30 September-2 October,
University of NSW, Sydney.
David, P. (1990), “The dynamo and the computer: an historical
perspective on the modern productivity paradox”,
American Economic Review, pp. 355-61.
David, J.S., Schuff, D. and St Louis, R. (2002), “Managing your IT
total cost of ownership”, Communications of the ACM,
Vol. 45 No. 1, pp. 101-6.
Dier, D.H. and Mooney, J.G. (1994), “Enhancing the evaluation of
information technology investment through
comprehensive cost analysis”, Proceedings of the 1st
European Conference for IT Evaluation, 13-14 September,
Henley Management College, Henley on Thames, UK.
Dirks, P. and Van Lent, L. (1997), “Control of the information
systems function: the role of the cost allocation”,
Proceedings of the 4th European Conference on IT
Evaluation, University of Delft, Delft.
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
DIST (1998), Building for Growth. A Draft Strategy for the
Building and Construction Industry, Department of
Industry, Science and Tourism, Commonwealth of
Australia Publication, Canberra.
Farbey, B., Land, F. and Targett, D. (1993), IT Investment: A Study
of Methods and Practices, Published with Management
Today and Butterworth-Heinemann, London.
Galliers, R.D. (1997), “Reflection on information systems: twelve
points to debate”, in Mingers, J. and Stowell, F. (Eds),
Information Systems: An Emerging Discipline, McGraw-
Hill, Berkshire, pp. 113-36.
Ghoneim, A., Irani, Z. and Love, P.E.D. (2003a), “Identifying and
classifying IS life cycle costs”, Association for Information
Systems, Americas Conference on Information Systems
(AMCIS 2003), 9-11 August, New Orleans, LA.
Ghoneim, A., Irani, Z. and Love, P.E.D. (2003b), A Review of
Indirect Cost Taxonomies for Information Systems Projects,
The 14th Australasian Conference on Information Systems
(ACIS’ 2003), Perth.
Ghoneim, A., Irani, Z. and Love, P.E.D. (2003c), “IT/IS evaluation:
the cost identification dilemma”, International Business
Information Management Conference (IBIM) on
E-Business and Organizations in the 21st Century,
16-18 December, Cairo.
Gummerson, E. (1998), Qualitative Methods in Management
Research, Sage, Newbury Park, CA.
Hides, M.T., Irani, Z. and Love, P.E.D. (2000), “Supporting
investment evaluation and project management:
framework of IT/IS costs”, International Conference
on Systems Thinking in Management (ICSTM-2000)
Deakin University Waterfront Campus, Geelong,
pp. 240-4.
Hochstrasser, B. (1992), “Justifying IT investments”, Conference
Proceedings: AIS, The New Technologies in Today’s
Business Environment, pp. 17-28.
Irani, Z. and Love, P.E.D. (2001), “The propagation of technology
management taxonomies for evaluating investments in
manufacturing resource planning (MRPII)”, Journal of
Management Information Systems, Vol. 17 No. 3,
pp. 161-77.
Irani, Z., Love, P.E.D. and Li, H. (1999), “IT/IS investment barriers
to the decision-making process”, Proceedings of the
Business Information Technology: The Global Imperative
Conference, 30 June-2 July, [CD-Proceedings] The
Department of Information Systems at the University of
the Western Cape, Cape Town.
Irani, Z., Love, P.E.D. and Hides, M.T. (2000), “Investment
evaluation of new technology: a conceptual model for
managing IT/IS cost”, paper presented at the Americas
Conference on Information Systems (AMCIS 2000),
[CD-ROM Proceedings], Long Beach, CA, 10-13 August.
Jick, T.D. (1979), “Mixing qualitative and quantitative methods:
triangulation in accumulation”, Administrative Science
Quarterly, Vol. 24, pp. 602-11.
Kaplan, R.S. (1984), “Yesterday’s accounting undermines
production”, Harvard Business Review, Vol. 62 No. 4,
pp. 95-101.
Kaye, G., Hinton, C. and Armstrong, S. (1995), “The hidden
investments in IT: dependency, legacy and heritage”,
Proceedings of the 2nd European Conference on IT
Investments Evaluation, pp. 249-59.
Kusters, R.J. and Renkema, T.J.W. (1996), “Managing IT
investment decisions in their organizational context: the
design of local for local evaluation models”, Proceedings
of the 3rd European Conference for IT Evaluation,
Bath University School of Management, Bath University,
Bath, UK.
324
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
Latham, M. (1994), Constructing the Team: Joint Review of
Procurement and Contractual Arrangements in the UK
Construction Industry, Department of the Environment,
London.
Lefley, F. (1994), “Capital investment appraisal of manufacturing
technology”, International Journal of Production Research,
Vol. 32 No. 12, pp. 2751-6.
Li, H., Love, P.E.D. and Irani, Z. (2000), “The relationship between
the use of IT/IS and the productivity of consulting firms in
construction”, International Journal of Construction
Information Technology, Vol. 8 No. 1, pp. 15-27.
Love, P.E.D. and Gunasekaran, A. (1997), “Process re-
engineering: a review of enablers”, International Journal
of Production Economics, Vol. 50 Nos 2/3, pp. 183-97.
Love, P.E.D., Gunasekaran, A. and Li, H. (1998), “Putting an
engine into re-engineering: toward a process based
organisation”, International Journal of Operations and
Production Management, Vol. 18 No. 9, pp. 937-49.
Mende, M.W., Brecht, L. and Osterle, H. (1994), Evaluating
Existing Systems from a Business Process Perspective,
Virginia.
Milis, K. and Mercken, R. (2004), “The use of the balanced
scorecard for the evaluation of information and
communication technology projects”, International
Journal of Project Management, Vol. 22, pp. 87-97.
Mohamed, S. and Irani, Z. (2002), “Developing taxonomies of
information system indirect human costs”, paper
presented at the International Conference on Systems
Thinking in Management, University of Salford, Salford,
3-5 April.
Neuman, W.L. (1991), Social Research Methods: Qualitative and
Quantitative Approaches, Allyn and Bacon, Boston, MA.
Remenyi, D., Michaek, S. and Terry, W. (1996), “Outcomes and
benefits for information systems investment”,
Proceedings of the 3rd European Conference for IT
Evaluation, Bath University School of Management,
Bath University, Bath.
Renkema, J.W. and Berghout, E.W. (1997), “Methodologies for
information systems investment evaluation at the
proposal stage: a comparative review”, Information and
Software Technology, Vol. 39, pp. 1-13.
Ryan, S.D. and Harrison, D.A. (2000), “Considering social
subsystem costs and benefits in information technology
investment decisions: a view from the field on anticipated
payoffs”, Journal of Management Information Systems,
Vol. 16 No. 4, pp. 11-40.
Shaughnessy, J.J. and Zechmeister, E.B. (1994), Research
Methods in Psychology, 3rd ed., McGraw-Hill, New York,
NY.
Stoeckerl, A. and Quirke, D. (1992), “Services: setting the agenda
for reform”, Report No. 2, Services Industries Research
Program, Department of Industry, Technology and
Commerce, Canberra.
Stirroh, K.J. (2001), “Investing in information technology:
productivity payoffs for US industries”, Current Issues in
Economics and Finance, Vol. 7 No. 6, June, Federal Bank of
New York, New York, NY.
Wheatley, M. (1997), “Hidden costs of the humble PC”,
Management Today, pp. 52-4.
Weill, P. and Broadbent, M. (1998), Leveraging the New
Infrastructure: How Market Leaders Capitalize on IT,
Harvard Business School Press, Boston, MA.
Willcocks, L. (1996), “Introduction: of capital importance”,
in Willcocks, L. (Ed.), Information Management: The
Evaluation of Information Systems Investments, Chapman
& Hall, London, pp. 1-27.
 Information technology evaluation: classifying indirect costs
Peter E.D. Love, Ahmad Ghoneim and Zahir Irani
Willcocks, L. and Lester, S. (1996), “The evaluation and
management of IS investments: from feasibility to routine
operations”, in Willcocks, L. (Ed.), Investing in Information
Systems: Evaluation and Management, Chapman & Hall,
London.
Yellow Pages (2002), Yellow Pagesw E-business Report: The
On-line Experience of Small and Medium Enterprises,
Yellow Pages Business Index Small and Medium
Enterprises, Australia, July.
Yin, R.K. (1989), Case Study Research – Design and Methods,
Applied Social Research Method Series, Vol. 34, Sage
Publications, Newbury Park, CA.
Further reading
Alter, S. (1999), Information Systems: A Management
Perspective, 3rd ed., Addison-Wesley, Reading, MA.
Andresen, J. (2000), “The unidentified value of IT in construction
industry”, Proceedings of the International Conference on
Information Technology in Construction: Implementing IT
to Gain a Competitive Advantage in the 21st Century,
17-18 January, Department of Building and Real Estate,
Hong Kong Polytechnic University, Hong Kong, pp. 93-105.
Ballantine, J. and Stray, S. (1999), “Information systems and
other capital investments: evaluation practice compared”,
Logistics and Information Management, Vol. 12 Nos 1/2,
pp. 78-93.
Berghout, E.T. and Renkema, T.J. (2001), “Methodologies for
investment evaluation: a review and assessment”, in Van
Grembergen, W. (Ed.), Information Technology Evaluation
Methods and Management, Idea Group Publishing,
Hershey, PA, pp. 26-43.
Checkland, P. (1995), “Soft systems methodology and its
relevance to information systems”, in Stowell, F. (Ed.),
Information Systems Provisions: The Contribution of Soft
Systems, McGraw-Hill, London, pp. 1-17.
Debras, P., Moneceyron, J-L., Bauer, F. and Ballesta, P. (1998),
“An information server for the building industry: interest of
a STEP/IFC based approach”, Proceedings of 2nd
European Conference on Product and Process Modelling in
325
The Journal of Enterprise Information Management
Volume 17 · Number 4 · 2004 · 312–325
the Building Industry, 19-21 October, Building Research
Establishment Ltd, Watford.
Hogbin, G. and Thomas, D.V. (1994), Investing in Information
Technology: Managing the Decision-making Process’,
McGraw-Hill/IBM Series, Cambridge.
Irani, Z. and Love, P.E.D. (2002), “Developing a frame of
reference for ex-ante IT/IS investment evaluation”,
European Journal of Information Systems, Vol. 11 No. 1,
pp. 74-82.
Irani, Z., Ezingeard, J.N. and Grieve, R.J. (1997), “Integrating the
costs of an IT/IS infrastructure into the investment
decision-making process”, The International Journal of
Technological Innovation, Entrepreneurship and
Technology Management (Technovation), Vol. 17
Nos 11/12, pp. 695-706.
Irani, Z., Love, P.E.D., Sharif, A.M. and Li, H. (2001), “The failure
and success of a manufacturing information system: a
case study”, European Journal of Information Systems,
Vol. 10 No. 1, pp. 55-66.
Lefley, F. and Sarkis, J. (1997), “Short-termism and the appraisal
of AMT capital projects in the US and UK”, International
Journal of Production Research, Vol. 35 No. 2, pp. 341-55.
Primrose, P.L. (1991), Investment in Manufacturing Technology,
Total Technology Department, UMIST, Manchester,
Chapman & Hall, London.
Rivard, H. (1999), A Survey of Information Technology in the
Canadian Construction Industry, in Lacasse, M.A. and
Vanier, D.J. (Eds), National Research Council of Canada,
Vancouver, pp. 2339-47.
Small, M.H. and Chen, J. (1995), “Investment justification of
advanced manufacturing technology: an empirical
analysis”, Journal of Engineering and Technology
Management, Vol. 12 Nos 1/2, pp. 27-55.
Silverman, D. (1998), “Qualitative research: meanings or
practice?”, Information Systems Journal, Vol. 8 No. 1,
pp. 3-20.
Strassmann, P. (1992), “The politics of downsizing”,
Datamation, pp. 106-10.
Stratopoulos, T. and Dehning, T. (2000), “Does successful
investment in information technology solve the
productivity paradox?”, Information and Management,
Vol. 38, pp. 103-17.